Mostly the “man” part. Pretty clear in the OP I thought. I was quite simply born as a male, and happened to identify as that gender. A significant enough portion of the population seems to believe that, because a patriarchy exists, all men have benefited from it, and all men want to continue it. The same idea plays through well enough for skin color, and orientation.
I know what I am, I know my thoughts, my feelings and my intentions. It starts to play with your sense of self-worth to be told that these things, things that have never caused you to do anything to harm anyone else, must be bad parts of yourself, because look at what people have done in their name.
It’s not the same scale, no. I’m not facing segregation, and don’t have to fight for my right to vote. Any of a number of other advantages you want to point out. Yeah, I benefited in some ways from the circumstances of my birth. All of this, common talking points from the sides of the aisle that I want to belong to. The side of the aisle that believes that no person should ever feel marginalized because of something that they had no control over. To hear that, and then feel like these same people are telling you you’re part of the problem because of your existence… It’s not hard to see how that can really impact one’s sense of worth to the world.