Amouranth made a chatbot clone to outsource flirting — and protect herself

there is... a lot going on here--and it's part of a broader trend which is probably not for the better and speaks to some deeper-seated issues we currently have in society. a choice moment from the article here on another influencer doing a similar thing earlier this year, and how that went:

Siragusa isn’t the first influencer to create a voice-prompted AI chatbot using her likeness. The first would be Caryn Marjorie, a 23-year-old Snapchat creator, who has more than 1.8 million followers. CarynAI is trained on a combination of OpenAI’s ChatGPT-4 and some 2,000 hours of her now-deleted YouTube content, according to Fortune. On May 11, when CarynAI launched, Marjorie tweeted that the app would “cure loneliness.” She’d also told Fortune that the AI chatbot was not meant to make sexual advances. But on the day of its release, users commented on the AI’s tendency to bring up sexually explicit content. “The AI was not programmed to do this and has seemed to go rogue,” Marjorie told Insider, adding that her team was “working around the clock to prevent this from happening again.”

aranym, (edited )

This strikes me as very exploitative. Capitalizing on loneliness to enrich yourself gives me bad vibes (especially when the users of this thing will actually be worse off mentally in the end, as @cnnrduncan mentioned).

I have no doubts that for some users, it’ll turn into a cycle. They’ll feel lonelier each time they use it, which pushes them to use it more, and so on. Amouranth does nothing to protect these vulnerable users. I had the same feelings about Replika years before ChatGPT became a thing.

Kaldo,
@Kaldo@beehaw.org avatar

She's doing the same thing as always, taking advantage of vulnerable people while playing herself off as a victim. This is hardly news or something positive about AI chat bots...

TenGallonRat,

I don't agree with Amouranth's statement of "I don’t think it’s harmful for people to form a social aspect with it. As long as we’re aware that it’s not a real person." You can tell people it's not real all you want, but when you model it off your voice/likeness, you're inherently attaching yourself to it. I'm sure the majority of people will understand it's an AI modeled off her voice, but there will always be some that get more attached to it than they should. Just saying "I don't think it's harmful because people should know it's not real" doesn't magically solve the issue and cause everyone to behave.

Replika is another smutty type AI, but it's clearly designated as such, and the avatar is pretty generic looking, which is more of what I'd want to see, at least for now. It being text-only also helps keep that separation. Who knows, it could end up fine, but this is closer to diving in the deep end before people have waded in to see what the water is like.

SilentStorms,

Yeah, and people got extremely attached to their Replikas. To the point that the subreddit stickied a suicide hotline number when the app removed sexual roleplay.

authorinthedark,

The slippery slope of "I know it's not real but we're so close and it's based off of you so therefore you should love me too" does not feel like that much of a stretch

jmcs,

Exactly. Celebrities already have a problem with people forming unhealthy parasocial relationships with them when the interactions are unidirectional, having something answering back is going to make it much, much, worse. Hopefully no one is going to be seriously hurt in the aftermath of this money-grubbing stunt.

Mindless_Enigma,
@Mindless_Enigma@beehaw.org avatar

I don't really like the logic of "chatbots like this will help cure loneliness." It might help someone feel less lonely at first. But then it'll be a crutch and, if anything, hurt people's ability to socialize with other real people. Like it's a quick dopamine hit that will slowly dig you deeper into the hole you feel you're in.

Parsnip8904,
@Parsnip8904@beehaw.org avatar

It is like any of these people have not seen any sci-fi released over the last 50 years.

cnnrduncan,

Adding on to your comment, I read a really interesting study recently that suggests that interacting with AI engages the social parts of our brains but does not provide the same stimuli/feedback as interacting with a real person leading to increased loneliness and thus increased alcohol abuse and insomnia.

Gaywallet,
@Gaywallet@beehaw.org avatar

Fascinating, do you happen to have a link? No worries if it's buried or you don't remember, but I'd love to read it

cnnrduncan,

Sure thing, here ya go!

Gaywallet,
@Gaywallet@beehaw.org avatar

Apologies for the extremely late reply but thank you so much for this fascinating article!

Noumena,

I was wondering. Seemingly, it could teach interaction skills as long as the bot wasn't abnormally accomindating. Maybe it works for some and not others. So, it could be a therapy option. Just needs to be monitored for whether the person uses it to hurt themselves.

PenguinTD,

Yeah, right, I don't buy it.

DarkDarkHouse,
@DarkDarkHouse@lemmy.sdf.org avatar

Enough people will.

thepiguy,

the conversation got even smuttier very quickly

lol I love the future.

briongloid,
@briongloid@aussie.zone avatar

Imagine paying to talk to their chatbot, thinking it's them.

maynarkh,

The AI was not programmed to do this and has seemed to go rogue

The AI does exactly what it was programmed to do. It's a machine. They fucked up making it, that's the reality here.

Mindless_Enigma,
@Mindless_Enigma@beehaw.org avatar

The cynic in me says they didn't actually fuck up. The AI being horny brought more attention to it and playing it off like "oh no! it went rogue!" gives them even more headlines. All a strategic play.

elessar,

@alyaza @maynarkh This storied of LLMs gone rogue is just a cover to not be held accountable for the failures of the machines.

mistersheep,

Yeah, I laughed at that part. The bot does what it interprets from the training data. Her entire schtick is leaning into sexual overtones to keep the guys subscribed. The fact that the bot started replicating that is entirely expected.

The trick there is that there is a fine line she walks between keeping it overt enough to keep the guys interested, but not so forward as to cross the terms of service boundaries on Twitch/YouTube. I think the problem is really that the bot isn't adhering to the "tone it down so we don't get banned from the service" unwritten rule. I'm not sure if I'd call that "rogue" though.

But, this is a great use case for a LLM, and if she can capitalise on it then good on her.

Gaywallet,
@Gaywallet@beehaw.org avatar

Eh, all chat bots trained on internet data become horny in the same way that virtually all AI is racist. It's merely a reflection of the data it's trained on, but a lot of people think they can magically control it by putting 'safeguards' after it's been trained. The constant game of cat and mouse with gpt3/4 and prompts to getting it to ignore safeguards are proof this doesn't work.

You can fix this through additional weighted layers and more complex upstream training tweaks, but then you have to pay to retrain which is by far the most expensive part about open AIs model.

a_statistician,

You can also curate the training data so that it's not problematic, but then you are biasing the model in other ways.

Gaywallet,
@Gaywallet@beehaw.org avatar

Impossible at scale. Data is always biased, it's a matter of tradeoff

loops,

It's like Her but with exceptionally more horni.

ag_roberston_author,
@ag_roberston_author@beehaw.org avatar

When I watched it, I really didn't understand just how prescient Her was going to be. Great movie, I should rewatch it.

Powderhorn,
@Powderhorn@beehaw.org avatar

Definitely a situation where the demand is more of a damning indictment than the fact that supply exists.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • technology@beehaw.org
  • DreamBathrooms
  • ngwrru68w68
  • modclub
  • magazineikmin
  • thenastyranch
  • rosin
  • khanakhh
  • InstantRegret
  • Youngstown
  • slotface
  • Durango
  • kavyap
  • mdbf
  • ethstaker
  • megavids
  • osvaldo12
  • tester
  • GTA5RPClips
  • cubers
  • everett
  • tacticalgear
  • cisconetworking
  • normalnudes
  • anitta
  • provamag3
  • Leos
  • JUstTest
  • lostlight
  • All magazines
    •