Heat the oil in a large Dutch oven or other heavy-bottomed pot. Add the shallots and fennel and cook until shallots are softened, about 3 minutes. Stir in the garlic, rosemary, and fennel seed and cook 1 minute longer. Stir in salt, pepper, and cayenne. Add the wine and bring to a boil. Stir in the mussels, cover, and cook until mussels have opened about 5 to 7 minutes, stirring occasionally. Transfer mussels to a deep serving bowl with a slotted spoon. Discard any mussels that do not open.
Stir in the butter, lemon juice, and lemon zest. Bring to a simmer and taste for seasoning. Pour over the mussels, and garnish with the chopped fronds and fennel pollen if using. Serve with the bread.