Understandable. I don't view showing downvotes as weakening filters—I see how it could deter downvoting to some extent, but in my experience, it leads to people handing out downvotes less freely and when it's more justified as opposed to not downvoting at all. Nevertheless, I see where you're coming from, and I want to stress that I think it's completely fair to not want people to see if you've downvoted them.
Note: I don't think defederating from an instance prevents them from seeing your content (according to this thread), so I'm not sure if defederation would make your votes private.