This is going to help scammers a lot. Imagine getting a call from your loved ones, and they explain they are in trouble and need immediate financial assistance. This is going to affect so many elderly and innocent people. OpenAI is developing these tools again from publicly scrapping of your voice samples.
Remember, Microsoft is investing another $100 billion into this company to build the giant AI supercomputer. They are equally responsible for this kind of problem. If this money were invested in cancer research, nobody would have said anything, but everything they do will either enable scammers or steal jobs.
I've actually personally witnessed a scam like this 4-5 years ago, minus the ai, with an individual pretending to be someone's grandson. Their basic ruse was that they'd been on vacation in Mexico and had gotten arrested because some friends they were riding with had marijuana and needed bail money wired. Call quality was bad, but understandable. Call came in to the business line of a u-store storage locker company. I played along, which I found quite amusing in retrospect.
You basically had a circumstance where the scammer needed to reach someone old with hearing problems and a lack of good sense. Might take a while, but it's a numbers game.
With AI voice imitation, the hit rate is going to be much higher, particularly if you can document relationships via social networks, cold call the grandson/granddaughter and obtain enough audio to produce a AI doppleganger. Makes me very wary of answering the phone with anything other than a simple hello.
@dany_57987@nixCraft If you use Win10 or 11, you have given Microsoft permission to use your microphone and listen in on you at any time without your knowledge.
It's part of the EULA.
@nixCraft
One measure is to establish a password with one's peers. I have one with my wife and kids. If we can't tell each other the password (e.g. the response is "what the hell do you mean, you have to help me right now!!"), we'll consider it fraudulent and drop the connection.
@nixCraft I mean, this already exists with 11labs and some other AI voice generators, but the question is how much more realistic is OpenAI's version, plus if it can display emotion (which could definitely help with scams)
@nixCraft I thought about that, but I do think it's going to sound off to the recipient, because you can't use it to have a conversation, you can only use it to create recordings. so it may work for a voicemail, but not for an actual phone conversation or something alike.
@anthropy@nixCraft people are already pulling this scam off now, without voice copying. But they have to stick to further relatives — grandchildren, especially.
This just makes the scam more widely usable.
And if you think people won’t fall for it, you don’t know enough about how scams work.
@lkanies@nixCraft I never said you can't scam people even with stupid non-voice scams, I never said this isn't a problem, I just said you're not going to be able to have a conversation with this thing because it just generates samples and takes time doing so.
@anthropy@nixCraft there’s no link to the original article, but I assumed it would be connected to the chat functions, so it was generating real time audio.
Even without that, voicemail would become a scam vector (and already is for lots of other scams).
So sure, not being able to be real time reduces the problem for a while. But it’s still a huge increase in danger, without thinking at all about the consequences.
@lkanies@nixCraft I don't think it's generating realtime audio, at best it's a chat where you have to wait X seconds before you get a response, so not exactly realtime. and again, I never said there wasn't an "increase in danger", I literally said "I also thought this", I merely added that it isn't real time or usable for realtime applications.
this didn't have to be an offense/defense kind of conversation, and honestly I'm not planning to get into one either, so I'll leave it at this.
@nixCraft It's awful, but also that scam has been going on for decades without AI. Meaning, the evidence of how people will for sure abuse this is there they just don't want to look at it.
But of course, this scam is more prominent in lower income countries or countries with a lot of violence and, I mean, eeww, right?
@ErikJonker@nixCraft It might be a good time to look back at more physical interactions - imagine the world's tightest weapons systems could be easily tricked by this.
Add comment