What kind of relationship is suggested if someone calls a girl "khon suay"? Is that a casual thing?

Khun Suay!

"You are Beutiful"

Or perhaps คนสวย - beautiful person...

if my fiance called his ex that, should I be worried?

คนสวย is a flirtatious term of endearment, in the same vein as "baby", "sexy", "cutie" etc. For example เป็นไง คนสวย "What's up, beautiful?"

I would say it brings your fiance's commitment to you into question. Keep an eye on him.

If he said that TO her, you might watch out. If he said that ABOUT her, then no big deal. It simply means pretty, could as easily apply to a soccer kick as a person...