There is one way of dealing with neighbors like this, that can be very effective.
Tell him that you want to be a good neighbor and would be willing to have the tree trimmed by a proffesional...if he would pay the bill.
Don't worry though, he won't, but if he does, I'm sure you could make the tree look good and reduce the size so it interfers less with the neighbor.
Good luck.