Preheat the oven to 350 degrees F (175 degrees C). Grease a 9x13-inch baking dish.
Heat oil in a skillet over medium heat; stir in onion and celery. Saute until the onion has softened and turned translucent, about 5 minutes. Season with salt and pepper, if desired. Transfer to a large bowl with whole-kernel corn, creamed corn, and crushed crackers.
Whisk together eggs and milk in a small bowl and add to the corn mixture. Add butter, pepper, and salt; mix until all ingredients are incorporated. Pour into the prepared baking dish.
Bake in the preheated oven until edges are golden brown, 45 to 60 minutes. Let cool for 5 to 10 minutes before serving.