The big difference is that the Fediverse, like Reddit and Digg before it, are not communication platforms between individual that know each other IRL.. They are content sharing and discussion platforms. The content that mainstream Meta or Twitter users are interested in and generating is largely not what I am interested in, so how is it a bad thing if most of them disappear from my platform?
Plenty of us have already gone through the most painful part of the transition and are now focused on building something new. If we can do that once, we can do it again, but it'll be even easier to divorce from Meta if I don't care about what I'm "losing" in the split.