Heat olive oil in a saucepan over medium heat. Cook and stir onion and celery in the hot oil until translucent, about 5 minutes. Add garlic and salt, and stir for 1 minute. Stir in potato, cauliflower, chicken broth, and water. Bring to a boil, reduce heat to medium-low and simmer until cauliflower is tender and potatoes are easily smashed against the side of the pot, about 30 minutes.
Puree the soup using an immersion blender until smooth. Reduce heat to low, season with cayenne pepper and salt. Pour in cream and stir until warmed. Season with more salt if needed.
Cook bacon in a skillet over medium heat until crisp, about 8 minutes. Transfer cooked bacon to a paper towel lined plate to drain. Toss breadcrumbs in the remaining bacon grease in the skillet, and cook over low heat until toasted and crisp. Stir in cheese and continue to cook until melted. Transfer to a bowl and cool to room temperature. Stir in bacon, lemon zest, and parsley. Toss to coat.
Serve soup garnished with bacon gremolata over the top.