Pastor Rick Stuckwisch has an excellent post talking about how hard it is when the people that he pastors make assumptions about him and fail to do the Christian thing, to talk with him about things that are bothering them that he might or might not have done.
As a pastor's wife, I've experienced that, but as much as that bothers me, I can vouch for the fact that it hurts more to see when it happens to my husband, knowing how much he really cares for his flock, how much he wants peace and to help, and prays daily for the welfare of his flock. What hurts the most is when a person or a family leaves in such a way that he has no idea why, and treats the relationship that they have like it is nothing. - when he has baptized and catechized their children, held their hands and prayed with them through crisis or even death.
It isn't pleasant to go to someone and say "what you did/said offended me," or "I don't like this." Christ makes it clear that we are supposed to do this -- we are to go to the person with whom we have a problem. We're not supposed to talk to others about it. That only hurts the person more, and also may bring the other person (people) into your problem. If you are complaining to another person - not with the one you have the problem, its gossip, and it is hurtful. You have no idea how much.