Bring milk to a boil and remove from heat. Add tea bags and steep for 30 minutes. Strain out tea. Squeeze out as much liquids from the tea as possible. Discard tea bags. Measure remaining liquid. Add additional milk as necessary until you have 1 cup of liquid. Set aside.
Preheat oven to 350 degrees F. Butter three 6-inch round cake pans. Line with parchment paper and set aside.
In the bowl of stand mixer fitted with paddle attachment, combine 2 cups flour (reserve remaining 1/2 cup flour), sugar, baking powder, salt, cinnamon, ginger, cardamom, and cloves. Add butter in three additions. Continue to mix on low speed until mixture is crumbly like coarse sand.
In another bowl, whisk together tea infused milk, eggs, and vanilla until smooth. Add half of liquid mixture to stand mixing bowl. Mix on medium speed until incorporated. Add remaining liquid mixture and mix until batter is combined. The batter will look curdled. Add reserved 1/2 cup of flour and mix until batter comes together.
Evenly distribute batter among prepared cake pans. Bake for 30-35 minutes until toothpick inserted in center of cake comes out clean. Allow cake to cool in pan for 10-15 minutes. Run an offset spatula along the sides of the cake pan to loosen cake. Invert cake onto a wire rack and allow to cool to room temperature.