Very similar to driver resistance to rolling down the window, not wanting to give change without having an exact fare, or wanting driving directions after the location is agreed upon. It's basically a powertrip by the driver.
The best thing to do is have the driver stop and find another taxi. You can fight them on it, but it makes the trip uncomfortable, and can cause further problems depending on how stubborn the driver is.