No problem. And I can imagine that there might be some out there who would indeed be gleeful about being called out for downvoting because they're just that deep into the "raging asshole" state that any negative attention is giving them the dopamine hit they crave. But I suspect that kind of troll is going to be blatant enough that he'll get blocked or banned by most places worth hanging out in anyway.
I probably shouldn't admit it, but one of my favourite ways of dealing with a raging asshole on Reddit was to be impeccably polite to them until they blew their stack badly enough while trying to provoke me that I could report them. :) I'd only do that if they were clearly already unredeemable, though.