In a large mixing bowl, whisk together the instant pudding mix, cold milk, cinnamon, nutmeg, ginger, and vanilla extract until smooth and thickened, about 2 to 3 minutes. Set aside.
In a separate bowl, beat the heavy whipping cream, powdered sugar, and vanilla extract until stiff peaks form. You can also use store-bought whipped topping if you prefer a quicker option.
To assemble, place a layer of broken shortbread cookies in the bottom of a trifle dish or individual serving cups. Add a layer of sliced bananas, followed by a layer of pudding. Sprinkle in some crushed gingerman cookies, then add a layer of whipped cream. Repeat the layers until all ingredients are used, finishing with whipped cream on top.
Garnish with crushed shortbread cookies, crushed gingerbread cookies, or a light dusting of cinnamon. Refrigerate for at least 1 hour before serving so the cookies soften slightly and the flavors come together.