At the moment I personally am a fan of the limited downvote federation because I do think it acts as a hivemind barrier;
I think it's a lot less straightforward than this. While it definitely drowns out the "hivemind" of the wider fediverse, it also creates a bubble within your own instance. If a lot of people outside of your instance think you made a bad comment but few inside your instance do, limited downvote federation creates an inaccurate representation of what people think.
Additionally, having downvotes but limiting their federation makes them extremely unintuitive and only serves to further confuse new users. It's actively misleading—you'd think that if you see 0 downvotes that nobody clicked the downvote button, but that might actually be 5 people or 10 people or 30 people. At least if you don't have a downvote counter at all, it's clear that downvote functionality just isn't recognized on the instance. I'd much prefer that over a straight up incorrect counter.
Overall though, I think the problem is how the downvote was used (on Reddit, at least) was not conducive to discussion.
I agree that there's a big problem with how downvotes are used. I personally use downvotes if something is:
spam
straight up hateful
completely off-topic
a nonsensical take that the poster doesn't support
In other words, if something isn't a reasonable contribution to the discussion, I'll likely downvote it.
However, lots of people seem to use the downvote button as a disagree button. I see this less on kbin.social than I do on other instances, and that could be a potential reason to not federate downvotes. If downvotes are used differently on kbin.social than on other instances, then I can see it making sense to not lump them all together. However, that only makes sense if people on kbin.social aren't using the downvote button as a disagree button (which they are, just less so than on other instances). And even still, there's the problem of an unfederated downvote counter still being misleading.
Rambling aside, I see three ways of handling this:
Remove the downvote button entirely.
Remove the downvote button, but replace it with reactions that people can use ("This is spam," "This is hateful", "This doesn't contribute to the discussion," etc.). If other instances adopt this set of reactions, reactions from those instances could federate.
Keep the downvote button and federate downvotes.
I don't like #1 very much because then you don't have a great way of indicating spam, hateful posts, unhelpful comments, etc. outside of reports that only moderators can see. #2 seems really nice, though both it and #1 come with the downside of filtering out negative feedback from instances with downvotes. #3 is also good, though it comes with the problems of downvote misuse.