The quality of the moderation and the tolerance of the community also plays a large part. A lot of what is tolerated on sites like reddit doesn't have to be on the fediverse (paradox of tolerance), making the place much less toxic.
OTOH, a large community makes moderation much more strenuous, which will end up being problematic.
There's probably a balance, and a tipping point after which it's no longer worth the effort. Needless to say we're way below that point.