As Mania mentioned, it would be very risky and dangerous for you if a physical fight ensued.
He has friends with guns under their motorbike seats, it'd be nothing to him to have you out of the picture, he's not worried about going to jail for that.
TIT ....
If you get into an altercation he's likely to call the police and they will side with the Thai .... you could end up in jail and charged with assault if they believe his story and not yours.
Move .... as far away as you can.