Preheat the oven to 350 degrees F (175 degrees C).
Mix sugar and butter together in a bowl using an electric mixer. Add eggs and beat to combine. Mix in sweet potatoes. Beat in salt, cinnamon, and nutmeg. Pour in milk and orange juice; mix very carefully until combined. Pour into a casserole dish.
Bake in the preheated oven until firm, about 45 minutes.