I understand what you are saying here, and I have flip-flopped on the issue myself. At the moment I personally am a fan of the limited downvote federation because I do think it acts as a hivemind barrier; to use reddit as an example, as it got bigger the downvote was used as a means to disagree without adding any value to the discussion or to simply silence a dissenting point of view.
That is not to say there were not times when a downvote was warranted - hateful comments, racism, sexism, homophobia, transphobia, and general bigotry are all more than deserving of a downvote. There are also bots and general off-topic posts & comments that may warrant a downvote.
Overall though, I think the problem is how the downvote was used (on Reddit, at least) was not conducive to discussion. However, due to the fact that upvotes and downvotes are public on kbin it is possible that behavior could change, but then that could create problems with other instances where none of that information is public to begin with. (Nobody wants to have a crazy person come after them over a downvote.)
Right now, the fediverse is pretty small and Kbin is actually the most welcoming instance I have found so far. I am not sure if the lack of downvote federation has anything to do with this, but so far I actually like it. Maybe once kbin and the expanded fediverse grows larger my opinion will change, but right now I feel like it’s helping to make it more hospitable than reddit.