Anyone know if there is any passenger boat on a regular basis to get there ? What's the timings, fare etc ?

There are no passenger ferries that go to Koh Phai. What you could do is take the regular ferry out to Koh Larn and then negotiate with a speed boat to run you out to Phai and pick you up at a later time. Of course you'd need to have a way of ensuring that they actually came back. Don't know if mobiles work as far out as Phai.