The man bag (or whatever it is called) is quite convenient IMHO.
The only downside is that you might leave it somewhere and forget. The simple solution for that, is to make it a habit never to take it off your body.
Driving, sitting, etc for long periods, with your wallet, often over stuffed, in the hip pocket may lead to muscle imbalances and nerve irritation in your hips and back.