My son has dual British/Thai nationality.

He entered Thailand on his Thai passport which has now expired,

Can he leave Thailand on his British one without renewing his Thai passpiort?

I would think not as it was never stamped on entry, It won't take long to get a new passport issued at your local office for the usual fee + any surcharge for express processing if urgent.

As your son entered Thailand with his Thai passport he should now get a replacement for his expired passport in Thailand. It will take only a few days. He cannot leave with his UK passport.

Applying for a Thai passport in Thailand is a breeze and very easy. It will be one of the easiest bureacratic experiences you'll have anywhere in the world. Three day turn around and they can send it to you via courier for an extra 30 baht.

He should obtain a new Thai passport but expect using the UK passport would not involve more than an overstay stamp on exit (and some explaining). The normal use of UK passport without entry stamp is for those born here and that is quite common.

was the expired Thai passport issued here?

Get a new Thai passport for him, as it will only take a few days. It will require both parents to sign the application form if under 21.

Leaving on the Thai passport will create problems as a didn't entered on his British passport and if he wants to use his Thai passport next time he enters Thailand there is no record he ever left Thailand.