I don't think it's as simple as that. Downvotes do have some kind of meaning — when you give a downvote, you're doing it for some reason that you want to convey — and people are going to interpret downvotes accordingly. If downvotes didn't mean anything, then there'd be no point to them existing at all. What exactly a downvote means depends on the person giving it, but it's ideally (imo) used to express that a post is spam, hateful, or otherwise a bad contribution to the discussion. Obviously, people shouldn't take downvotes personally, but a post being downvoted does and should mean something.
Thus, what OP mentions in his post is a legitimate concern. Public votes allow people to more easily downvote spam someone who downvoted them, which is unequivocally a bad thing that we'd prefer not to have. However, whether we should make votes private is a matter of whether the downsides outweigh the upsides, and they don't.