Preheat the oven to 350 degrees F (175 degrees C).
Steep Earl Grey tea bag in hot milk in a bowl.
Mix sugar and butter together in a large bowl. Mix in egg. Remove tea bag from hot milk; add milky tea mixture to batter. Add flour, baking powder, baking soda, and lavender; mix dough until well combined.
Drop dough by rounded spoonfuls onto ungreased cookie sheets.
Bake in the preheated oven until edges are light brown and set, 10 to 12 minutes. Let cool 10 minutes.
While cookies are cooking, stir lemon juice and water together in a small bowl. Add sugar, lavender, and lemon zest and stir until combined. Brush or spoon glaze onto cooled cookies.