This story has a easy solution. Everytime he comes by, tell him you're on your Way to a church meeting. To learn about the saving Grace of Jesus Christ, and he's more than welcome to come with you. If that doesn't have him running in the opposite direction, ask your wife how she would feel if you went golfing or name your entertainment of choice while she had to entertain this dude by herself. If she doesn't realize that he's a couple of tomatoes shy of a thick sauce now, maybe she will after a few hours alone with said individual.
Good luck and btw of course don't leave your wife alone with him. And secondly this is not intended to offend Christian as I'm one, I just know that most non-believers want nothing to do with the subject of Jesus Christ. If he does want to go with you you better know how to find a Church and fast.