You're right of course. People will flock to Meta, it will probably become the poster boy of the Fediverse over a few years, and then little by little the evil will creep in until it's so established we just accept it, same as we've done with Facebook. The terrible thing is that it will not be something we can just op-out of. I can chose not to use Facebook. With this situation, I would have to chose not to use the entire ActivityPub protocol, not just Meta's platform.
It's a disaster waiting to happen. Like you said, I don't think we can do much, and even if we try, it'll fracture the whole fediverse concept. But when you ask "Why are people concerned about Meta using ActivityPub?" this is why.