By this measure you must agree then that Budapest Pride should go ahead, as I detailed in a previous post that 73% of Budapest residents are against banning it. It clearly doesn't offend them as it's a huge majority.
Moreover, majorities aren't always morally right. Most Germans supported Naziism.
Your "fire" analogy is a false equivalence as that's civil disorder and possibly a crime which free assembly isn't.
Minority rights are moral and equality issues and shouldn't be beholden to the misguided. There are plenty of issues that received popular support in the past that we now see as morally reprehensible.
And it's not about getting absolute freedoms but retaining equality of freedoms.