I feel this way too. I know nearly who calls me ma’am is intending to be courteous and I don’t hold it against them. That said, knowing they are well intended doesn’t make me less uncomfortable.
Also the idea of sir being the term of respect for all men and even boys but ma’am being for “older” women adds some baked in unavoidable sexism, no matters how genuinely-not-actually-sexiest the speaker is. There are just necessary built in assumptions about the addressee when you have to choose between ma’am and miss (or similar). The implication is that societal value of women, and not men, is age-determined. The former often makes a woman feel undesirably old and the latter often makes her feel infantalized. It’s the same as the Mr./Mrs./Miss situation, where moving just to Mr. and Ms. alleviates that tension a bit. No clear answer for sir and ma’am honorifics though.