You are probably seeing two very different vocal minorities, and conflating the two.
Also, there’s a very clear difference in expectations between posting/commenting and upvoting. I blame the UI. We naturally expect public actions to be easily visible. The lack of universal accessibilty to the public data makes people unaware that the data is public. Lemmy UIs, including apps, need to make this information (a list of upvoting users) universally publicly accessible before people will change their expectations.