I am going to try getting to Ban Phe (Koh Samet pier link) via Rayong's bus terminal. Is it easy enough to tell them "Ban Phe" or can it be harder at these less touristed bus terminals?

I was there last week. When you walk in they should ask you where you want to go. Otherwise there are drivers in the main outside area who will ask. You should be ok. I guess you are saying terminal 2. There are two but I have never been to no 1