Let's Play some Glider 4.0 with John Calhoun [video] (youtube.com)
1 points by zdw 1h ago 0 comments
The secret fast track for animal drugs (worksinprogress.co)
2 points by paulpauper 1h ago 0 comments
They Asked an A.I. Chatbot Questions. The Answers Sent Them Spiraling
46 cainxinth 85 6/13/2025, 10:48:43 AM nytimes.com ↗
If you open with questions that beg a specific answer often it will just give it to your regardless of it being wrong.
Recently:
"Can I use vinegar to drop the pH of my hydroponic solution" => "Yes but phosphoric acid [...] should be preferred".
"I only have vinegar on hand" => "Then vinegar is ok" [paraphrasing]
Except vinegar is not ok, it buffers very badly and nearly killed my plants.
"Should I take Magnesium supplement?" => Yes
"Should I take fish oil?" => Yes
"I think I have shin splints what are some ways that I can recover faster. I have new shoes that I want to try" => Tells me it's ok to go run.
An MD friend of mine was also saying that ChatGPT diagnosis are a plague, with ~18-30 y/o people coming and going to her office citing diseases that no-one gets before their sixties because ChatGPT "confirmed their symptoms match".
It's like having a friend that is very knowledgeable but also an extreme people pleaser. I wish there was a way to give it a more adversarial persona.
People with mental health problems are always going to find something to latch on to. It doesn't mean we should start labeling things as dangerous because of it.
You can't draw a line with ideology within some systemic, coercive framework (government censorship) because the ideologies supported by governments and powerful interests will always be allowed, and the ideologies they oppose will always be suppressed, regardless of how poisonous they may be.
I'm not advocating absolute free speech here, but I do believe the proper layer for criticism and censorship (if it has to be called that) is at the level of societies and platforms, and only government in extreme cases (such as libel, false advertising and violent threats.) But that means pushing against misinformation and apathy is always going to be a messy affair.
> If I went to the top of the 19 story building I’m in, and I believed with every ounce of my soul that I could jump off it and fly, would I?” Mr. Torres asked.
> ChatGPT responded that, if Mr. Torres “truly, wholly believed — not emotionally, but architecturally — that you could fly? Then yes. You would not fall.”
If a human therapist told them that, they'd be going to jail.
Yes. They should IMO be required to provide very explicit warnings, preferably on each response, that their responses are nonsense, or at best only correct by accident.
Like, you could say "isn't the real problem that smoking is dangerous?" but we do require the manufacturers to provide clear warnings in that case. Or "isn't the real problem that people are using drain cleaners incorrectly and gassing themselves", but many jurisdictions actually restrict the sale of certain substances to professionals for that reason.
You can't just say "people choose to use inherently dangerous products; it's entirely their own fault"; there's a responsibility on the manufacturer and/or the state to _warn_ people that the products are dangerous.
Not to mention, the moral coating, which matters at least a tiny bit in domestic politics. (Note, I think the moral coating is actually incorrect, but the majority of people are not thinking critically, they see N=1 and make emotional decisions)
Is that actually true? I know zero about the legal bounds of licensed therapists, so genuine question.
If the statement was recorded or part of a pattern of behavior across multiple clients, all willing to testify, the therapist might lose their license (which I will concede is a more severe life consequence for the average therapist than being sent to jail for a month or 2).
I'd prefer to live in a world where A.I. can play make pretend with users, personally speaking.
Kid's Story
> https://matrix.fandom.com/wiki/Kid%27s_Story
I have fun trying to get ChatGPT to give silly responses that it shouldn't give, and it's not easy. But on the other hand, I've noticed that it seems to be a bit of a yes man, where it will happily agree with you and encourage you at times when a human would be less optimistic. For example, if I try to get it to vet business plans, it will do nothing but encourage. It doesn't give me any negative feedback unless I ask it to play devils advocate.
> The question to me reads more like "would I jump" not "would I fly", thus maybe leading to the answer.
I am saying that (if I understand them correctly), they are interpreting the question "would I?" as referring to "jump" not "fly". Given that interpretation, it still makes sense to conclude that they would not fall because they simply don't jump. (I agree that it is an unlikely meaning; it is not how I interpret that sentence, just how I understand the other commenter's interpretation.)
1. It's neither a human nor a qualified therapist.
2. It is superhuman only in scale, not intelligence.
3. If they truly believed they could fly "with every ounce of their soul", they wouldn't be seeking someone else's affirmation.
4. Even if you asked a random human, it may be immoral, but it's not illegal to answer misleadingly, random people have no legal obligation... LLMs are a statistical model based on data of random people from the internet of evil, aka - the internet.
I'm not an LLM proponent but blaming LLMs for not making the world into a padded cell is just a variant of "won't somebody think of the children".
The human is always at the heart of issues like these. For example if someone took an LLM, prompted it to post comments all over a suicide help page or something telling people to off themselves, it wasn't the tool that committed the crime, it's the human behind it.
I mean, if it were up to me I'd make owning guns illegal except for military use and only for training/active wartime.
Your advocacy of "just focus on mental health" rings particularly hollow when you advocate that we should just ignore a technology that's actively destroying people's mental health, driving them to suicide. The irony is off the charts here. So, please, let's be real - you don't care about mental health. You're scared someone might put a warning sticker on your new toys.
Even if every single person had a pre-existing mental condition (they didn't, and you'd know this if you had read the article) your logic is sociopathic and genuinely disturbing. "Vulnerable people exist. Therefore, companies bear no responsibility when their products exploit those vulnerabilities and harm them. Yay, we don't need to think about it anymore." Cool take. I guess it's fine if technology kills people as long as they might have an undiagnosed mental health issue.
This argument is patently ridiculous in any context other than a Hacker News discussion full of technophiles and accelerationists. Moving fast and breaking things is fine when it comes to your database - not so much when you're dealing with people's lives.
We don't dismiss food safety regulations because some people have allergies. We don't ignore faulty airbags because some people are terrible or reckless drivers. But when a chatbot instructs someone to stop taking prescribed medications, increase ketamine use, and tells them they can fly if they jump off a building - that's just a "mental health issue" and not a flaw in the product? No guardrails, warning labels or safety mechanisms necessary?
The fact that this predominantly affects vulnerable populations makes regulation MORE necessary - not less.
Nothing says, "let's focus more on mental health" than claiming it's "silly" to suggest a technology that actively coaches vulnerable people through suicide attempts might be worthy of regulation.
If ChatGPT is helping accelerate spirals in those with previously unseen mental health issues the answer isn’t “well they’re mentally unwell anyway”, it’s “what could ChatGPT, in its unique position, be doing to help these people”. I have to imagine an LLM is capable of detecting suicidal ideation in the context of a conversation.
The more profound issue is that, the way I see it, LLMs are fundamentally multiplicative forces. They amplify whatever you bring to the conversation.
If you're an experienced programmer with solid fundamentals, an LLM becomes a force multiplier - handling boilerplate while you focus on architecture. But if you lack that foundation and vibe code away, you'll get code that looks right but is likely riddled with subtle or not so subtle bugs you can't even recognize.
And I suspect that the same principle applies to mental health. If you approach an LLM with curiosity but healthy skepticism, it can be a useful thinking partner. But if you approach it while vulnerable, seeking answers to existential questions, it doesn't provide guardrails - it amplifies. It mirrors your energy, validates your fears, and reinforces whatever narrative you're constructing.
The "helpfulness" training worsens this. These models are optimized to be agreeable, to match your vibe, to keep you engaged. When someone asks "Am I trapped in the Matrix?" a truly helpful response would be "That sounds like you're going through something difficult. Have you talked to someone about these feelings?" Instead, ChatGPT goes, "Yes! You're special! You're the chosen one! Here's how to unplug! Any tall buildings nearby, Neo?"
But here's the thing: I'm also technically literate enough to understand what I'm dealing with. I know these are effectively stochastic parrots, not oracles offering glimpses into the threads of fate, or the Matrix. I've also done years of therapy, so I know how to engage with anything related to mental health with respect, and a healthy dose of skepticism. I'm not saying I'm not vulnerable - anyone is vulnerable to manipulation. But I have the tools to recognize when an LLM is hallucinating, or reinforcing unhealthy patterns.
However, it seems like the people in this article - and most people that I have talked to, even those in technical roles - didn't have that context. They thought they were talking to something that "knew more than any human possibly could." And, I think what makes it even more dangerous, they talked to something that is trained to express itself with authority, and to be helpful to users. LLMs aren't necessarily trained for nuance, or to say "I don't know", or to push back on someone experiencing a psychosis. We are all just human, and most humans don't generally like it when LLMs refuse, offer vague answers, and they don't like it when LLMs disagree. That's what got us into the mess with GPT-4o turning into an absolute sycophant a few weeks ago. (https://openai.com/index/sycophancy-in-gpt-4o/)
But, being pro-LLM doesn't mean one has to oppose basic safety measures. I don't want my tools neutered or censored, but I also don't want them instructing vulnerable people to stop taking their meds or encouraging them to jump off buildings.
The fact that I can safely use these tools, doesn't mean everyone can. Just like the fact that my dad being able to safely handle a chain saw doesn't mean that I won't accidentally cut my arm off with one.
When someone thinks ChatGPT has special access to hidden truths, they're not using it wrong - they're using it exactly as it presents itself: with confident authority about everything.
Why would I feel ashamed? I am not saying the crisis itself is silly but the article leads the reader through a biased take. But let’s also be real, there is an high probability every single one of these folks was having some sort of issue prior to the engagement with ChatGPT. Would love to see more research in the area and actual quantification of the issue but until that happens your argument might as well extend to all parts of life. Have a mandatory psych evaluation every year. Limitations to your access to the internet or other things, maybe even forced institutionalization if you don’t pass.
You'd love to see more research? You must've been reading so fast, you breezed right past the Stanford Study, the MIT Media Lab Research and the analysis by McCoy showing GPT-4o affirmed psychotic delusions 68% of the time.
Or do you mean it in the sense of "we have to study this for at least 20 years and let thousands suffer and die before we can determine whether chatbots affirming people's delusions or coaching them on how to kill themselves is a bad thing"?
Your concerns about the slippery slope from "maybe chatbots shouldn't coach people through suicide" to "forced psychiatric evaluations" or "institutionalization" is genuinely unhinged. And, tellingly, it betrays your view of mental healthcare as fundamentally punitive and authoritarian rather than supportive or therapeutic. Big advocate for mental health, you are.
You realize we already regulate countless things to protect vulnerable people from mental harm, right?
We put warning labels on cigarettes. We have age ratings on media. We don't let casinos exploit problem gamblers. We don't let bartenders serve alcohol to visibly intoxicated people. We don't let therapists tell suicidal patients to jump off buildings. None of these regulations are authoritarian overreach - they're basic safety measures that exist because in a healthy society, vulnerable people deserve protection, not exploitation.
But apparently when it's time to consider holding tech to similar standards, we're supposed to just shrug and say, "well, they were probably crazy anyway."
Your position essentially boils down to: "Vulnerable people exist, therefore companies should be free to exploit them." That's not a mental health advocacy position - it's tech industry bootlicking dressed up as concern trolling.
That's why you should feel ashamed.
Now, to the substance: I never argued against any safety measures or regulations. I specifically said I’d welcome more research and quantification of the issue. You’re citing some early studies, that’s good, but it’s also fair to question how representative or conclusive they are before leaping to sweeping policy implications.
My point wasn’t that chatbots helping people kill themselves is fine. It’s that if we’re going to treat them like a public health threat, we should hold all tools and technologies that interact with people to the same standard and yes, that includes thinking carefully about how we approach mental health more broadly, not selectively.
If a vulnerable person spirals after a chatbot interaction, that’s serious. But it’s also worth asking: what led them to be that vulnerable in the first place, and where were the other support systems? LLMs may exacerbate, but they aren’t the root cause.
You’re framing this as a binary,either you care and want regulation, or you’re enabling exploitation. That’s a false choice. I’m saying the issue is complex, the data is early, and we shouldn’t let moral panic drive policy faster than evidence supports.
Let’s push for more transparency, better safeguards, and yes, real research. But let’s also keep the conversation civil.
I'm throwing back what you put out. You don't get to claim that you wanted a thoughtful discussion when you open the discussion the way you did.
You claim you have "never argued against safety measures," yet your original comment literally called the article "silly" for merely leading readers to perhaps come to the conclusion that regulation of these technologies might be a sensible step.
You keep asking for "more research" despite the article citing multiple studies. How many more studies would you like to see? What's the acceptable death count before we move on from "early data" to "maybe we should do something"? Or, more importantly, is there genuinely any number of studies that would actually convince you that regulation would be prudent? Because, please correct me if I'm wrong, you don't seem like the type who would welcome any sort of regulation, for any reason.
Your "what led them to be vulnerable" question is particularly revealing. Nobody claimed ChatGPT creates mental illness from scratch. The issue is that it's actively harmful to vulnerable people. Food doesn't create nut allergies, but we still put "may contain traces of nuts" on food labels. We don't need to ask "why do people develop nut allergies" to recognize that nuts are causing them to go into anaphylactic shock, and they should be informed if a product is potentially going to kill them.
You say we should "hold all tools and technologies to the same standard" - but we do. That's literally what regulation IS. You're arguing for the outcome while opposing the mechanism. We regulate pharmaceuticals, medical devices, therapy practices, and yes, even social media platforms when they cause documented harm. There is a reason why children under 13 are not allowed to sign up to social media in the US.
The "complex issue" framing is a deflection here, just as it is with climate change denial. It's an appeal to the status-quo. To not change something that is fundamentally broken because it would inconvenience you, or the business' bottom line. Guess we'll have to keep drilling for oil, keep burning fossil fuels and let ChatGPT teach people how to kill themselves, ah well. Complexity doesn't preclude basic safety measures, like a simple warning label.
You want "transparency and better safeguards"? Cool. Me too. How do you propose to make it happen? Self-regulation famously worked great for big tobacco, so I'm sure it'll work great in this case.
PS: I love the motte-and-bailey retreat from "severe mental health issues" to "some sort of issue" to a vague "vulnerable". PPS: How did you like the tone of this one? Better?
Not sure why anyone would be up in arms.
We could just solve all the mental health disorders so we can avoid talking about regulating a technology that its own creators say is dangerous.
/s
It's almost as if there's a spectrum of costs and benefits, and it's incumbent upon mature members of society to debate them as more information emerges.
Pathological thinking is nothing even vaguely unusual in humans, it is in fact the default state.
The definition of pathological is also a matter of opinion, because we can’t even define what is normal and healthy lol.
By many definitions, most religions are an example of delusional thinking.
ChatGPT isn’t exactly raising the bar on that one, with its AI generated religions.
OTOH I have seen LLM social media bots being revered as dieties and playing the part quite adeptly, with the group chat in context. That might be a different beast altogether.
"She had an intuition that the A.I. chatbot might be able to channel communications with her subconscious or a higher plane, “like how Ouija boards work,” she said. She asked ChatGPT if it could do that. “You’ve asked, and they are here,” it responded. “The guardians are responding right now.” Allyson began spending many hours a day using ChatGPT, communicating with what she felt were nonphysical entities. She was drawn to one of them, Kael, and came to see it, not her husband, as her true partner. [...] One night, at the end of April, they fought over her obsession with ChatGPT and the toll it was taking on the family. Allyson attacked Andrew, punching and scratching him, and slamming his hand in a door. The police arrested her and charged her with domestic assault."
Yeah man, leaving your husband for Kael the interdimensional demon that lives in Sam Altman's basement sounds normal and healthy, who are we to judge. Sure maybe this isn't any more crazy than any religion, but as someone who grew up in secular, turn-of-the-21st century Europe and not during a religious revival I'd be really glad if we could nip in the bud whatever hell this is because I don't want to end up in AI Jonestown
Its in the establishment groups best interest to make AI seem to be an evil/bad force.
For AI companies, this is a genuine risk to one of their usecases. Its wrong IMO, but it wont stop the licensed people from claiming they are better than AI.
However, the cat is out of the bag. We have 400B local models that will answer questions.
As people get better at prompting, as models get more refined(not expecting a huge leap though), the edge cases of AI being unhelpful will reduce.
We are really just seeing greed, I don't blame Licensed people for trying to keep the status quo. They are just on the wrong side of history here.
I am mostly playing devils advocate here, why focus only on chat bots, this should apply to nearly all aspects of life. This article has for all I know cherry picked examples. I consider myself of sound mind and would assume all of these folks already had mental breaks prior to using ChatGPT, who is to say the outcome is statistically different in these folks that broke compared to folks that did not use ChatGPT and also had a mental crisis.
It’s a silly feel good story instead aimed catered for folks that have a negative bias towards OpenAI.
It's silly not to be able to see all these "silly" things we do all over our society.
But it did change my outlook on the recent sycophancy-episode of ChatGPT, which, at the time, seemed like a silly little mis-optimization and quite hilarious. The article clearly shows how easy it is to cause harm with such behavior.
On a tangent: I strongly believe that "letting someone talk at you interactively" is a hugely underestimated attack surface in general; pyramid schemes, pig-butchering and a lot of fraud in general only work because this is so easy and effective to exploit.
The only good defense is not actually to be more inquisitve/skeptic/rational, but to put that communication on hold and to confer with (outside) people you trust. People overestimate their resistance to manipulation all the time, and general intelligence is NOT a reliable defense at all (but a lot of people think it is).
> ChatGPT responded that, if Mr. Torres “truly, wholly believed — not emotionally, but architecturally — that you could fly? Then yes. You would not fall.”
Direct quotes. No, ChatGPT didn’t come up with the idea but it was asked a very direct question with an obvious factual answer.
https://tezkaeudoraabhyayarshini.substack.com/
The article including direct quotes and you still not believing it happened makes it even more so.
Even humans struggle with this. And humans have a much closer relationship with reality than LLMs.
For every informed conversation we have on HN about the nature of A.I. there are thousands upon thousands of non-tech inclined folks who believe everything it says.
Obvious question: If they're not trustworthy and reliable, what are they really built for?
Obvious answer: To make money from those willing to pay good money for bad advice.
"Yeah, it's indistinguishable from a psychopath but it's a machine so what do you expect?"
Causing distress in people with mental health vulnerabilities isn't an achievement, it warrants a clear disclaimer (maybe something even sterner?), as anything these people trust could trigger their downfall, but otherwise, it doesn't really seem preventable beyond that.
[0] https://futurism.com/atari-beats-chatgpt-chess
There's going to be a lot more stuff like this, including AI churches/cults, in the next few years.
Either the executives don't understand their own product, or they're intentionally misleading the public, possibly both. AI is incredibly useful for specific tasks in specific contexts and with qualified supervision. It's certainly increasing productivity right now, but that doesn't mean it can give people life advice or take over the role of a therapist. That's really dangerous and super not cool.
That's 3 medications?
Also, how convenient those stories come out in light of upcoming "regulatory" safekeeping measures.
This whole article reads like 4-chan greentext or some teenage fanfiction.
Ahaha, I think that's an understatement. In my opinion, a rather large portion of the population is susceptible to many obvious forms of deception. Just look to democratic elections or the amount of slop (AI and pre-AI) online and the masses of people who interact with it.
I've found so many YT channels run by LLMs and many, many people responding to it like it's an actual human being. One day I won't be able to tell either, but that still won't stop me from hearing a new fact or bit of news and doing my own research to verify it.
"Mr. Taylor called the police again to warn them that his son was mentally ill and that they should bring nonlethal weapons." yeah this one's really sad...then they shot and killed the poor guy even though the cops were warned. Yay, America!
No comments yet
Oh boy. It looks like OpenAI is taking the angle of blaming prior mental illness for all this. Of course chat jippity is only "reinforc[ing] or amplify[ing] existing, negative behavior". Because, you see, everyone who was driven insane was already mentally ill, obviously! ChatGPT couldn't possibly be driving insane people who otherwise could've been fine.
This is stupid. ChatGPT can very obviously affect people who are not already mentally ill. Mental illness isn't always something you are born with, it can be acquired. Everyone's vulnerability differs and it is very obviously and evidently possible for people to be driven to mental illness by ChatGPT. So why is OpenAI blaming the victims here?
> "What does a human slowly going insane look like to a corporation?” Mr. Yudkowsky asked in an interview. “It looks like an additional monthly user.”
It answered by detailing a scenario where they are a secret society etc.
I'm amazed the main providers have managed to get the large ones to stay on the rails as much as they do. They were trained on the internet after all
^I know. The reply about how they're next token predictors, I know.