While I personally never downvote anyone for disagreeing, I don't think it's possible, or desirable, to try to create rules around how people use their preference buttons.
But I def suggest going into your settings and hiding vote scores, as that's psychologically better for most people.