I useally smile, and back off, or face it, so far no one tried me.
Happened in Fortaleza Brazil, 3 men gave it a second thought in a side street I used as a short cut in the middle of the day, something I was told not to do. I was lucky that day, no consequences, just a heads up, always listening to the locals.