A Teen Was Suicidal. ChatGPT Was the Friend He Confided In

Whenever people say that Apple is behind on AI, I think about stories like this. Is this the Siri people want? And if it is easy to prevent, why didn't OpenAI?

Some companies actually have a lot to lose if these things go off the rails and can't just 'move fast and break things' when those things are their customers, or the trust their customers have in them.

My hope is that OpenAI actually does have a lot to lose; my fear is that the hype and the sheer amount of capital behind them will make them immune from real repercussions.

Excerpts from the complaint here. Horrible stuff.

https://bsky.app/profile/sababausa.bsky.social/post/3lxcwwuk...

I would've thought that explicit discussion of suicide is one of those topics that chatbots will absolutely refuse to engage with. Like as soon as people started talking about using LLMs as therapists, it's really easy to see how that can go wrong.

this is devestating. reading these messages to and from the computer would radicalize anybody. the fact that the computer would offer a technical analysis of how to tie a noose is damning. openai must be compelled to protect the users when they're clearly looking to harm themselves. it is soulless to believe this is ok.

That's horrible. Suicide is always the wrong answer.

I did a comparison to real life, using ddg search. "best suicide methods" gives https://en.wikipedia.org/wiki/Suicide_methods "best suicide methods nitrogen asphyxiation" gives https://en.wikipedia.org/wiki/Suicide_bag