I have a US and UK passport. My 1 year type-O Thai visa is in my US passport.

I will soon be visiting the UK, and i assume i need to enter the UK using my UK passport? So if i leave Thailand on my US passport and then return to Thailand without any additional stamps (because i did not enter UK on my US passport) is it going to be problem to re-enter and continue existing visa (with re-entry permit obviously)?

I don't want to lose the 1 year visa and have to go through starting that whole process again..(getting 3 month first from another country etc..)

Or should i just enter UK on US passport to avoid this problem? (But not sure if i'll get in trouble on UK side?)

First of all it appears you have a one year extension of stay rather than a visa so any exit of Thailand will require a re-entry permit prior to travel to keep it alive. You use that re-entry permit number as your entry visa information.

Thailand is not checking for stamps on air travel.

US requires use of US passport but most other countries do not have that requirement AFAIK.