All racial terms become derogatory over time. But at the time, it was OK for my gran to say 'golliwog' or 'negro', Thai people to say 'jek' for Chinese people. Both are now not OK, and the words for Indians, 'Kaek' has become derogatory, as has 'farang'.
I am a 'sweaty sock'. I take no offense to 'fat <deleted>' in Austin Powers or Ewan Mcteagle