From the perspective of a taxi driver, there is no downside to trying to renegotiate a fare en route to the destination. In his mind, the worst that can happen is that he'll get the originally agreed fare. So as a passenger, it's up to you to shift some of the risk back to him.
I did that once with a taxi driver in Cairo, Egypt, on my way from my hotel to the bazaar. We had agreed a fare of 10 Egyptian pounds. Halfway to the bazaar, he turned around and said to me "15 pounds!". After a brief moment of panic, I responded "five pounds". He just laughed - and said "OK, 10 pounds."
They need to learn that renegotiation can go both ways.