Assuming you're being sarcastic and mean the opposite, this hasn't been my experience, actually. Just like with @livus, above, I called out a downvote-stalker once who'd been following me around and when I described how I was seeing his downvote pattern he instantly vanished. In my experience the "downvote warriors" are a cowardly bunch, they love being able to throw punches without being seen to throw punches. Once you make it clear to them that everyone can see what they're doing they crumple under scrutiny.
The trolls you're talking about are the kind that love to get into an argument with you. That's quite different.