Upon further analysis it seems to be specific to certain people, not doormen/security in general. So my issue is with people who don't do security/lobby well, not just anybody. It could very well be me staring at them also, I don't know who is doing it first, we are both looking.
I looked back at places that really know what they are doing and it was not an issue. So for other readers it may be helpful to pop into a place you are renting or buying during the day and night to see if you're comfortable with the vibe. Some people call it the 'milieu'.
As for the place I'm at now, putting on my sunglasses a bit early helps a lot and makes a huge difference. I now know why rockstars wear sunglasses all the time, it's not just for the sun!