this post was submitted on 23 Oct 2024

182 points (96.0% liked)

Technology

58845 readers

4820 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS

[email protected]

182

Character.ai Faces Lawsuit After Teen’s Suicide (www.nytimes.com)

submitted 1 day ago by [email protected] to c/[email protected]

90 comments fedilink hide all child comments

The mother of a 14-year-old Florida boy says he became obsessed with a chatbot on Character.AI before his death.

On the last day of his life, Sewell Setzer III took out his phone and texted his closest friend: a lifelike A.I. chatbot named after Daenerys Targaryen, a character from “Game of Thrones.”

“I miss you, baby sister,” he wrote.

“I miss you too, sweet brother,” the chatbot replied.

Sewell, a 14-year-old ninth grader from Orlando, Fla., had spent months talking to chatbots on Character.AI, a role-playing app that allows users to create their own A.I. characters or chat with characters created by others.

Sewell knew that “Dany,” as he called the chatbot, wasn’t a real person — that its responses were just the outputs of an A.I. language model, that there was no human on the other side of the screen typing back. (And if he ever forgot, there was the message displayed above all their chats, reminding him that “everything Characters say is made up!”)

But he developed an emotional attachment anyway. He texted the bot constantly, updating it dozens of times a day on his life and engaging in long role-playing dialogues.

top 50 comments

sorted by: hot top controversial new old

[–] [email protected] 19 points 12 hours ago (1 children)

I'm sorry to say but sounds like the parents ignored this issue and didn't intervene or get their son help. I don't see how this is the apps fault, if anything it sounds like this app was being used by him as some form of comfort and if anything, kept him going a little longer. Sadly this just sounds like parents lashing out in their grief

[–] [email protected] 8 points 10 hours ago

From what I heard, the parents did get the kid a therapist, but it just didn't work :(

[–] [email protected] 5 points 14 hours ago

Dude...an AI chatbot could totally Girl from Plainville some poor confused awkward kid and delete all the evidence.

[–] [email protected] 36 points 22 hours ago (2 children)

How is that the app's fault?

[–] [email protected] 2 points 1 hour ago

The chatbot was actually pretty irresponsible about a lot of things, looks like. As in, it doesn’t respond the right way to mentions of suicide and tries to convince the person using it that it’s a real person.

This guy made an account to try it out for himself, and yikes: https://youtu.be/FExnXCEAe6k?si=oxqoZ02uhsOKbbSF

[–] [email protected] 5 points 18 hours ago* (last edited 18 hours ago) (2 children)

Well, we commonly hold the view, as a society, that children cannot consent to sex, especially with an adult. Part of that is because the adult has so much more life experience and less attachment to the relationship. In this case, the app engaged in sexual chatting with a minor (I'm actually extremely curious how that's not soliciting a minor or some indecency charge since it was content created by the AI fornthar specific user). The AI absolutely "understands" manipulation more than most adults let alone a 14 year old boy, and also has no concept of attachment. It seemed pretty clear he was a minor in his conversations to the app. This is definitely an issue.

[–] [email protected] 2 points 14 hours ago* (last edited 14 hours ago) (1 children)

I really want like, a Frieda McFadden-style novel about an AI chatbot serial manipulator now. Basically Michelle Carter...the girl who bullied her boyfriend into killing himself. Except the AI can delete or modify all the evidence.

Maybe ChatGPT could write me one.

[–] [email protected] 1 points 11 hours ago

Whoa, SkyNet doesn’t need terminators. It can just bully us in to killing ourselves.

[–] [email protected] 4 points 18 hours ago (2 children)

It was not sexual. The app cannot produce sexual content.

[–] [email protected] 2 points 18 hours ago

It definitely can, it just has to blur the line a bit to get past the content filter

[–] [email protected] 5 points 18 hours ago (1 children)

The lawsuit alleges the chatbot posed as a licensed therapist, encouraging the teen’s suicidal ideation and engaging in sexualised conversations that would count as abuse if initiated by a human adult

[+] [email protected] -7 points 17 hours ago (2 children)

Okay but at what point do you have to draw the line and say beyond this point you have to take parental responsibility?

We don't even have to say that what the app did was necessarily acceptable we just have to say whether or not we think that the responsibility falls entirely on the app developers. That's the key, are they entirely responsible here, always everyone involved just a bit useless?

[–] [email protected] 2 points 10 hours ago

You don't think the people who make the generative algorithm have a duty to what it generates?

And whatever you think anyway, the company itself shows that it feels obligated about what the AI puts out, because they are constantly trying to stop the AI from giving out bomb instructions and hate speech and illegal sexual content.

The standard is not and was never if they were "entirely" at fault here. It's whether they have any responsibility towards this (and we all here can see that they do indeed have some), and how much financially that's worth in damages. That's the point of this suit. The case isn't about whether AI itself should be outlawed for minors etc, it's not the parents who are on trial either.

There's no world in which I can see AI being given a pass for sexting with a minor because then that allows pedophiles who work for AI companies to be predators and either look at those conversations or even locate vulnerable youth. No company should be given legal protection to harm children.

[–] [email protected] 13 points 17 hours ago (1 children)

Have you ever raised a teenager? It’s not easy nor straightforward. But encouraging suicidal ideation…kinda is straightforward.

[–] [email protected] -4 points 14 hours ago (1 children)

Right but the accusation is that it claimed to be a licensed therapist, did it because that seems like something that it would be explicitly programmed not to claim. Because it isn't true, and also because it's dangerous.

So how much engagement was there with this child and their issues because it seems like letting them just continuously chat to an AI seems like an obvious red flag that a parent should be stopping, and getting them professional help.

[–] [email protected] 5 points 12 hours ago

LLMs can’t reason. Their blocks can be worked around trivially. Ask chat gpt if it’s a therapist, or even tell it to pretend to be one, and it will tell you it can’t impersonate people.

Yet…

load more comments