Preheat the oven to 325 degrees F (165 degrees C). Spray an 8-inch square baking dish with nonstick cooking spray. Place cubed bread inside of dish and set aside.
Bring milk, cardamom pods, allspice berries, cinnamon, and nutmeg to a simmer in a saucepan over medium-low heat; simmer for about 5 minutes. Remove from heat and let cool for 15 minutes. Strain milk into a bowl to remove large spices.
Heat 4 tablespoons butter in a saute pan over medium-high heat. Saute apple in the hot butter until tender, about 5 minutes. Turn off heat and add 1 cup brown sugar, vanilla extract, and caramel extract. Stir until sugar is melted. Pour over bread cubes and toss until evenly distributed.
Beat eggs and cooled, spiced milk together in a mixing bowl using an electric mixer. Pour over the bread mixture. Allow bread to soak up mixture, 10 to 15 minutes, lightly tapping down bread cubes as needed so egg mixture can soak in.
Sprinkle remaining 1/4 cup brown sugar over the bread pudding. Melt remaining butter in a microwave-safe dish in a microwave, 15 to 30 seconds. Drizzle over brown sugar.
Bake in the preheated oven until set and a toothpick inserted into the center comes out clean, 45 to 60 minutes. Serve warm or cold.