Place potatoes into a large pot and cover with water. Bring to a boil over high heat; reduce heat to medium-low, cover, and simmer until tender, about 20 minutes. While potatoes are boiling, mash garlic with 1/2 teaspoon of salt into a smooth paste. Drain potatoes, then mash together with garlic paste, butter, and pepper. Keep warm.
Preheat the oven to 500 degrees F (260 degrees C).
While potatoes are cooking, pulse hazelnuts and bread crumbs in a food processor until finely ground, but not ground into a paste. Pour onto a shallow dish. Season halibut fillets on both sides with 1/4 teaspoon of salt. Gently press one side of halibut fillets into nut mixture, and set aside.
Heat olive oil and 1 tablespoon of butter in an ovenproof skillet over medium-high heat. Place halibut fillets nut-side-up into the skillet; cook until halibut begins to brown, 3 to 4 minutes. Turn fillets over, and place into the preheated oven. Bake, nut-side-down until fish flakes easily with a fork, about 5 minutes.
Melt remaining 5 tablespoons of butter in a small saucepan over medium-high heat until it turns nut brown. Stir in vegetable broth, and bring to a simmer.
To serve, mound potatoes onto the center of each dinner plate. Place a halibut fillet on top of mashed potatoes nut-side-up, then drizzle with browned butter sauce.