Hmm you raise an interesting point. I do agree that it's helpful to explain to someone why they're being downvoted.
An experimental feature might be to allow downvotes only if you reply, or else you can choose to down vote a comment if you also upvote a response to the downvoted comment.