Line a 9-inch square baking pan (with a removable bottom is best) with parchment paper along the bottom and sides to allow for easy cake removal. Line the baking pan with a layer of graham crackers. Cut crackers as necessary to fit pan.
In chilled bowl, use a handheld mixer to whip heavy cream to soft peaks. Continue to whip while slowing adding sweetened condensed milk. Whip to firm peaks.
Add about 1 cup of sweetened whipped cream over graham crackers. Distribute about 1 cup of diced mango over whipped cream. Place a layer of graham crackers over mangoes. Repeat twice by adding another layer of whipped cream followed by mangos.
Cover pan with plastic wrap and let chill in the fridge for at least 4 hours, overnight is best. Alternatively, you can freeze the dessert for more of an ice cream cake kind of texture. Once chilled, lift mango royale out of the pan using the parchment paper overhang. Slice and serve.