Yes, I could see that a futile house visit would be irksome.
Simplistically, there is no legal reason why the OP could not undertake his trip - it could easily be business/work related. However, commonsense, and courtesy, dictates that a discussion with the IO takes place - and maybe supply an accompanying note explaining the necessity of the trip.