I think I get what you're saying, and there was a time when I would've agreed. I spent more years than I care to admit on 4chan, years I wouldn't have spent if I didn't think there was some value to people expressing their opinions no matter what they were. But...I dunno man, it's not a ton of people, but I wouldn't call it a "very small" number of people. Also the issue I'm getting at isn't that they have a platform, it's that if you let them they will try to make every platform their platform. And if it's an organized group they will do so in an organized way that is not the same as Uncle Ted cocking off about immigrants again or whatever.
You're correct that you don't have to look at their profile, any more than you have to drink the pube punch. The issue isn't that I had to see the words of meanies. The issue is that allowing white supremacists to use your platform a) makes it look like the platform condones such things, which reflects both on the platform and the other users, which may cause the non-extremist users to leave if it gets bad enough, this tipping the balance of users more in the extremists' favor; and b) encourages people who agree with them. And the number of people who think certain people shouldn't have rights doesn't have to be very big for them to decide to organize and do something about it, including egging others on.
Also you mentioned tweets, so I should apologize for not clarifying before. When expressing concern over extremists inviting themselves, I was not thinking about Twitter so much as I was thinking about the fediverse. I'm more concerned with what people are trying to build here than with whatever it is they're doing at Twitter these days. Elon's gonna Elon and we can't control that. We can, though, choose what company we keep here.