Well usually the people who get mad about it are the exact same people who want to participate in these communities. "Free speech" to them just means being a racist shithead on the internet without consequences and there is no reason that should be tolerated. Quite the opposite, we should strive to make these people feel as unwelcome as possible.