Add the milk and horchata to a large, high-walled skillet or saucepan over medium heat. Bring the mixture to a simmer, whisking constantly to make sure it does not burn. Add the sweetened condensed milk and whisk to combine. Reduce the heat to low.
Add the egg yolks to a medium bowl. While whisking constantly, slowly ladle the horchata mixture into the eggs to temper the yolks.
Slowly pour the egg mixture back into the pan and increase the heat to medium. Continue cooking, whisking constantly, until the mixture becomes thick and creamy, about 5 minutes.
Remove the pot from the heat and pour into a clean medium bowl. Place the bowl with the ice cream base in the ice bath, making sure the ice water doesn’t get into the base, and stir until chilled completely, about 20 minutes. Transfer the base to an airtight container, cover, and freeze for at least 2 hours, up to 1 week.
Scoop and serve. Garnish with whole cinnamon sticks and sprinkle with ground cinnamon, if desired.