I'm with you, and whilst I've referred a lot to downvoting as a means of disagreement in my comments here, in my instance, they weren't downvoted specifically because I just didn't like what was in them, I didn't think they were conducive to the topic at hand and were promoting 'redditification' already. If I could have supplied that as a reason for downvote, I absolutely would have.
I believe another user has mentioned Tildes system of being able to flag comments, and whilst I'm not a Tildes user myself, I've lurked the odd post, and think it's a decent system. Whether Ernest agrees is another question, and also the beauty of federation. Someone can fork it and make it do that (even if it is local only, not sure how that would work across ActivityPub).