Mix sugar, butter, egg, cinnamon, salt, and nutmeg together in a bowl. Stir in sweet potatoes, orange zest, and orange juice until evenly incorporated. Pour mixture into a shallow 9x13-inch baking dish.
Bake in the preheated oven until cooked through and bubbling, 1 hour. Serve hot or cold.