It would seem that many non-users don't like the smell of it wafting around public places. It's a 'pungent' and not very pleasant smell. In the UK recently a colleague complained that he could smell it when his neighbours were smoking it (in their back garden).
(I'm not virulently anti-drugs, I've had a smoke a few times before, partly out of curiosity, quite pleasant).