Make the banana custard: In a large bowl, mash the bananas. Add the eggs, milk, and heavy cream and stir to combine.
In a medium bowl, combine the sugar, cinnamon, nutmeg, and vanilla and mix with a fork.
Add the sugar mixture to the custard mixture and stir to combine.
Add the cubed bread, chocolate chips, and pecans to the custard mixture and toss well to coat. Let soak for 10 minutes.
While the bread is soaking, make the caramel base: In a medium microwave-safe bowl, microwave the butter for about 2 minutes, stirring every 30 seconds, until melted and hot.
Add the brown sugar and stir to combine. Microwave for another minute, stirring halfway, until the sugar is mostly dissolved.
Add the salt, vanilla, and pecans and stir to combine.
Pour the caramel into the bottom of a 9 (22 cm) x 13-inch (33 cm) baking dish and spread evenly. Arrange the bananas, cut side down, over the caramel.
Spoon the brioche-custard mixture evenly over the bananas.
Cover with foil and bake for 30 minutes, then uncover and bake for 40-45 minutes longer, until the custard is set. Remove from the oven and let cool for 5 minutes.
Run a knife around the edges of the baking dish to loosen the bake, then invert onto a rectangular serving dish. Do not wait any longer than 5 minutes or the bake will stick to the pan.