My missus and I (and latterly our daughter - now 11 yrs) have been visiting Koh Samed (or Ko Samet) on average about 2-3 times each year since 2001 - excluding the Covid years.
IMO there's not much to do on Samet, apart from lazing around, so a 3-4 day relaxing trip is optimum. Hire a scooter for the day - there's just one road around the island. Hire a canoe/jetski for a few hours and that's about it. Beach resorts have a BBQ and fire show in the evening, some have a disco until 2400hrs.
As a family attraction, that is what keeps us returning each year. We don't need the night life.
We have stayed at beach resorts all over the island but we now come back to Samed villa every time.
https://samedvilla.com/ - great buffet breakfast.
There are much more expensive options (we tried a couple, on the sunset side of the island) but IMO not worth the extra.
Of course YMMV.