Of course it's not fair that you have to do an hour of sweeping up every day. However, if you let this fester, it could spoil the rest of your day too.- IMO.
I think your "hardly the end of the world" attitude is the way to go.
Try to do the sweeping at a time that your neighbour will see it. Who knows, he may realise that he should take better care of his tree.
You did say that he was a nice enough bloke before you fell out.