I think some people have the setting on where when they interact with a post, that hides it from ever being shown to them again. And the quickest way to interact and hide something you don’t want to see would be the downvote button.
That would explain why it’s only ever a few downvotes here and there, compared to when someone posts something actually unpopular or offensive and it gets mass downvoted.