@HamSwagwich I think that's where I'm at too. Replace downvote with report button. If it's really irrelevant or otherwise breaking the rules let's remove it properly. If it isn't, well. It isn't, so let it stay without punishment.
Edit: Oooh, okay wait. Maybe let downvotes stay but make it benign. We're okay with casual upvotes for agreement. We should be okay with casual downvotes for disagreement. Its does let people see that their comment has been seen as is unpopular, compared with just unnoticed. I'm okay with that style of downvote being private.