Ok, I have a multiple entry tourist visa and need to leave the country, then get stamped in again.

Air Asia has a flight from Bangkok to Penang. When it reaches Penang, the same plane loads right up and returns to BKK, the turnaround time is less than an hour. This would be ideal for me (essentially, get off the plane then get back on). Problem is the AA check-in time in Penang: the flight will "close" before the plane from BKK arrives.

I haven't run this by AA yet, I'm planning on dropping by one of their offices later this week and see what comes of that. The answering I'm anticipating is "yes, you pay now" but when the day comes being told this is impossible.

Has anyone ever tried this, where when you check in you tell the AA clerk what you're doing, and things are set for you to re-board at the other end?

Air Asia is a point to point carrier. They have no 'transit' facilities. That means you will need to clear passport control/immigration on arrival at Penang and then check-in again for your return flight. As you say, check-in closes for the return flight before the incoming flight arrives - so it won't work.