So what happens with 300 people downvote a post and 500 upvote it? For that to work you'd need an 'account' per post/vote/user combination. Now your instance has 1000's of bot accounts that are now indistinguishable from bad vote manipulation.
ActivityPub broadcasts all these activity types. It's how it works. You can't federate these things without, well, federating them.