The downvote is useful to get rid of antivaxx"? You have a report button for that.
Yeah, but you have only so many mods that can do so much work. Having some degree of community "moderation" only serms sane to me.
Especially since every community will have its own preferences and interests so in that sense votes will also help to shape a community. Some might not like any memes in their group, some might etc etc