Place the instant vanilla pudding mix and cold milk in a large mixing bowl.
Whisk for about 2 minutes until smooth and thickened.
Add the vanilla extract.
Fold in the whipped topping until the mixture is light and creamy.
Spread a thin layer of the pudding mixture in the bottom of a large trifle dish.
Arrange half of the thawed mini cream puffs over the pudding.
Add half of the sliced bananas.
Drizzle half of the salted caramel sauce evenly over the bananas.
Sprinkle lightly with sea salt.
Spread half of the remaining pudding mixture over the caramel layer.
Repeat with the remaining cream puffs, bananas, caramel sauce, and pudding mixture.
Smooth the top evenly.
Sprinkle with crushed vanilla wafers or graham crackers if using.
Drizzle with a little extra caramel sauce if desired.
Cover the trifle.
Refrigerate for at least 2 hours to allow the layers to chill and set.
Remove from the refrigerator.
Serve chilled.