I’m sorry, but your assessment of how difficult that would be is WAAAAAY off.
Scammers are already doing stuff like this en masse with highly customized email scams.
The way this scam would work is to start with YouTubers, where grabbing the voice data is easy. Then you find their Facebook profile… Very easy, since people use the same usernames, or they go out of their way to link their profiles.
It’s a pretty easy step to make friend requests with those people. And then a very easy leap to find their relatives real names and towns through their Facebook connections.
Now you take their connections and towns and do reverse phone number lookups.
ALL of this can be automated. Every step.
The voice cloning and gpt-powered phone calls can be automated now, too.
The only reason this isn’t happening at scale is that scammers haven’t had enough time to adapt yet.