Absolutely right. What makes or breaks any social media platform is the ease of forming large communities (which goes hand in hand with the number of total active users) first and the user experience second.
"Fediverse" seems to suffer greatly from a UX point of view, mostly due to decentralization, which creates this isolation effect for newcomers.
Take mastodon vs twitter for example. For someone used to signing up for twitter and instantly gaining access to virtually everything the platform has to offer, mastodon has a big threshold to jump over before you can have a twitter-like experience. At least it feels like it until you get used to the experience. That's still the biggest barrier in front of large scale adoption of decentralized social media platforms.