In a large cast iron skillet over high heat, grill the ears of corn until charred on all sides, turning every 2 minutes, 10–15 minutes total.
Remove the corn from the pan, cut the kernels off the cobs, and return the cobs to the skillet. Char on all sides, turning every 2–3 minutes, 10–12 minutes total. Remove the cobs from the pan.
Cut the cobs in half crosswise and place in a small saucepan. Add ½ cup sugar, the milk, and cream. Bring to a gentle simmer over medium-low heat, 5–7 minutes. Transfer to a heatproof bowl or container and cover with plastic wrap or a lid. Steep in the refrigerator for 2 hours, or up to overnight.
Remove the cobs from the cream mixture, squeezing as much liquid from the cobs back into the bowl as possible.
Add the lemon and lime zest to a fine-mesh sieve set over a clean small saucepan. Carefully pour the cream mixture through the sieve over the zest, removing any corn bits that may remain. Bring to a simmer over medium-low heat.
In a medium bowl, whisk together the egg yolks and remaining ½ cup sugar. Using a measuring cup or ladle, slowly add the warm cream mixture to the eggs to temper, whisking constantly. Once all of the cream has been added, pour the mixture back into the pot and cook until thick enough to coat the back of a spoon, about 3 minutes. If desired, strain the mixture through the sieve once more to remove any curdled bits. Pour the mixture into a clean heatproof bowl or container and cover with plastic wrap. Chill in the refrigerator for at least 4 hours, up to overnight.
Transfer the ice cream base to a loaf pan or container and cover. Freeze for 4 hours, or up to overnight, until solid. (Alternatively, pour the ice cream base into an ice cream maker and mix according to the manufacturer’s instructions. Transfer to an airtight container and freeze until ready to serve.)
Scoop the ice cream into bowls and top with caramel corn and squeeze of lime.