In a medium saucepan, combine the water, sugar, and orange juice. Bring to a simmer over low heat. Cook, removing the pot from the heat and swirling if the bubbles begin to rise to the surface, until the mixture is reduced to a thick syrup and the color turns light brown, 20–30 minutes.
Remove the pot from the heat, add the butter, and stir immediately to combine. Add the cream, vanilla paste, allspice, cayenne, and salt and stir until incorporated and the consistency is smooth. Let cool for at least 10 minutes before using, or let cool completely before transferring to an airtight container and storing in the refrigerator until ready to use, up to 3 weeks.
Drizzle over ice cream and top with walnuts and orange zest, if desired, or serve with apple slices for dipping.