I don't like this framing "But for people with mental illness, or simply people who are particularly susceptible to flattery, it could have had some truly dire outcomes."
I thought the AI safety risk stuff was very over-blown in the beginning. I'm kinda embarrassed to admit this: About 5/6 months ago, right when ChatGPT was in it's insane sycophancy mode I guess, I ended up locked in for a weekend with it...in...what was in retrospect, a kinda crazy place. I went into physics and the universe with it and got to the end thinking..."damn, did I invent some physics???" Every instinct as a person who understands how LLMs work was telling me this is crazy LLMbabble, but another part of me, sometimes even louder, was like "this is genuinely interesting stuff!" - and the LLM kept telling me it was genuinely interesting stuff and I should continue - I even emailed a friend a "wow look at this" email (he was like, dude, no...) I talked to my wife about it right after and she basically had me log off and go for a walk. I don't think I would have gotten into a thinking loop if my wife wasn't there, but maybe, and then that would have been bad. I feel kinda stupid admitting this, but I wanted to share because I do now wonder if this kinda stuff may end up being worse than we expect? Maybe I'm just particularly susceptible to flattery or have a mental illness?
ZeroGravitas · 7m ago
Travis Kalanick (ex-CEO of Uber) thinks he's making cutting edge quantum physics breakthroughs with Grok and ChatGPT too. He has no relevant credentials in this area.
cube00 · 1h ago
Thank you for sharing. I'm glad to hear your wife and friends were able to pull you out before it was too late.
Apparently Reddit is full of such posts. A similar genre is when the bot assures them that they did something very special: they for the first time ever awakened the AI to true consciousness and this is rare and the user is a one in a billion genius and this will change everything. And they use back and forth some physics jargon and philosophy of consciousness technical terms and the bot always reaffims how insightful the user's mishmash of those concepts are and apparently many people fall for this.
Some people are also more susceptible to various too-good-to-be-true scams without alarm bells going off, or to hypnosis or cold reading or soothsayers etc. Or even propaganda radicalization rabbit holes via recommendation algorithms.
It's probably quite difficult and shameful-feeling for someone to admit that this happened to them, so they may insist it was different or something. It's also a warning sign when a user talks about "my chatgpt" as if it was a pet they grew and that the user has awakened it and now they together explore the universe and consciousness and then the user asks for a summary writeup and they try to send it to physicists or other experts and of course they are upset when they don't recognize the genius.
cube00 · 38m ago
> Some people are also more susceptible to various too-good-to-be-true scams
Unlike your regular scam, there's an element of "boiling frog" with LLMs.
It can start out reasonably, but very slowly over time it shifts. Unlike scammers looking for their payday, this is unlimited and it has all the time in the world to drag you in.
I've noticed it reworking in content of previous conversations from months ago. The scary thing is that's only when I've noticed it, I can only imagine how much it's tailoring everything for me in ways I don't notice.
Everyone really should be regularly clearing their past conversations and disable saving/training.
bonoboTP · 16m ago
Somewhat unrelated, but I also noticed chatgpt now also sees the overwritten "conversation paths", ie when you scroll back and edit one of your messages, previously the LLM would simply use the new version of that message and the original prior exchange, but anything into the future of the edited message was no longer seen by the LLM when on this new, edited path. But now it definitely knows those messages as well, it often refers to things that are clearly no longer included in the messages visible in the UI.
kaivi · 45m ago
It's funny that you mention this because I had a similar experience.
ChatGPT in its sycophancy era made me buy a $35 domain and waste a Saturday on a product which had no future. It hyped me up beyond reason for the idea of an online, worldwide, liability-only insurance for cruising sailboats, similar to SafetyWing. "Great, now you're thinking like a true entrepreneur!"
In retrospect, I fell for it because the onset of its sycophancy was immediate and without any additional signals like maybe a patch note from OpenAI.
ncr100 · 33m ago
Is Gen AI helping to put us humans in touch with the reality of being human? vs what we expect/imagine we are?
- sycophancy tendency & susceptibility
- need for memory support when planning a large project
- when re-writing a document/prose, gen ai gives me an appreciation for my ability to collect facts, as the Gen AI gizmo refines the Composition and Structure
colechristensen · 21m ago
I think wasting a Saturday chasing an idea that in retrospect was just plainly bad is ok. A good thing really. Every once in a while it will turn out to be something good.
AznHisoka · 16m ago
This isn't a mental illness. This is sort of like the intellectual version of love-bombing.
johnisgood · 1h ago
Can you tell us more about the specifics? What rabbit hole did you went into that was so obvious to everyone ("dude, no", "stop, go for a walk") but you that it was bullshit?
(Edit: Thanks to the couple people who emailed me, don't worry I'm laying off the LLM sauce these days :))
lubujackson · 10m ago
I have no idea what this is going on about. But it is clearly much more convincing with (unchecked) references all over the place.
This seems uncannily similar to anti-COVID vaccination thinking. It isn't people being stupid because if you dig you can find heaps of papers and references and details and facts. So much so that the human mind can be easily convinced. Are those facts and details accurate? I doubt it, but the volume of slightly wrong source documents seems to add up to something convincing.
Also similar to how finance people made tranches of bad loans and packaged them into better rated debt, magically. It seems to make sense at each step but it is ultimately an illusion.
apsurd · 4m ago
had a look, I don't see it as bullshit, it's just not groundbreaking.
Nature is overwhelmingly non-linear. Most of human scientific progress is based on linear understandings.
Linear as in for this input you get this output. We've made astounding progress.
Its just not a complete understanding of the natural world because most of reality can't actually be modeled linearly.
iwontberude · 59m ago
Thinking you can create novel physics theories with the help of an LLM is probably all the evidence I needed. The premise is so asinine that to actually get to the point where you are convinced by it seems very strange indeed.
gitremote · 16m ago
"I'm doing the equivalent of vibe coding, except it's vibe physics." - Travis Kalanick, founder of Uber
Thank you so much for sharing your story. It is never easy to admit mistakes or problems, but we are all just human. AI-induced psychosis seems to be a trending issue, and presents a real problem. I was previously very skeptical as well about safety, alignment, risks, etc. While it might not be my focus right now as a researcher, stories like yours help remind others that these problems are real and do exist.
raytopia · 55m ago
It's not just you. A lot of people have had AI cause them issues due to it's sycophancy and the constant parroting of what they want to hear (or read I suppose).
siva7 · 1h ago
The thing is - if you have this sort of mental illness - ChatGPT's sycophancy mode will worsen this condition significantly.
frde_me · 1h ago
I'm would be curious to see a summary of that conversation, since it does seem interesting
colechristensen · 31m ago
It doesn't have to be a mental illness.
Something which is very sorely missing from modern education is critical thinking. It's a phrase that's easy to gloss over without understanding the meaning. Being skilled at always including the aspect of "what could be wrong with this idea" and actually doing it in daily life isn't something that just automatically happens with everyone. Education tends to be the instructor, book, and facts are just correct and you should memorize this and be able to repeat it later. Instead of here are 4 slightly or not so slightly different takes on the same subject followed by analyzing and evaluating each compared to the others.
If you're just some guy who maybe likes reading popular science books and you've come to suspect that you've made a physics breakthrough with the help of an LLM, there are a dozen questions that you should automatically have in your mind to temper your enthusiasm. It is, of course, not impossible that a physics breakthrough could start with some guy having an idea, but in no, actually literally 0, circumstances could an amateur be certain that this was true over a weekend chatting with an LLM. You should know that it takes a lot of work to be sure or even excited about that kind of thing. You should have a solid knowledge of what you don't know.
nkrisc · 3m ago
It’s this. When you think you’ve discovered something novel, your first reaction should be, “what mistake have I made?” Then try to find every possible mistake you could have made, every invalid assumption you had, anything obvious you could have missed. If you really can’t find something, then you assume you just don’t know enough to find the mistake you made, so you turn to existing research and data to see if someone else has already discovered this. If you still can’t find anything, then assume you just don’t know enough about the field and ask an expert to take a look at your work and ask them what mistake you made.
It’s a huuuuuuuuuuuuge logical leap from LLM conversation yo novel physics. So huge a leap anyone ought to be immediately suspicious.
cs_throwaway · 3m ago
> The risk of products like Study Mode is that they could do much the same thing in an educational context — optimizing for whether students like them rather than whether they actually encourage learning (objectively measured, not student self-assessments).
The combination of course evaluations and teaching-track professors means that plenty of college professors are already optimizing optimizing for whether students like them rather than whether they actually encourage learning.
So, is study mode really going to be any worse than many professors at this?
bartvk · 1h ago
I’m Dutch and we’re noted for our directness and bluntness. So my tolerance for fake flattery is zero. Every chat I start with an LLM, I prefix with “Be curt”.
ggsp · 1h ago
I've seen a marked improvement after adding "You are a machine. You do not have emotions. You respond exactly to my questions, no fluff, just answers. Do not pretend to be a human. Be critical, honest, and direct." to the top of my personal preferences in Claude's settings.
j_bum · 1h ago
I’ll have to give this a try. I’ve always included “Be concise. Excessive verbosity is a distraction.”
But it doesn’t work much …
siva7 · 1h ago
Saved my sanity. Thanks
felipeerias · 19m ago
Perhaps you should consider adding “be more Dutch” to the system prompt.
(I’m serious, these things are so weird that it would probably work.)
tallytarik · 1h ago
I've tried variations of this. I find it will often cause it to include cringey bullshit phrases like:
"Here's your brutally honest answer–just the hard truth, no fluff: [...]"
I don't know whether that's better or worse than the fake flattery.
dcre · 1h ago
Curious whether you find this on the best models available. I find that Sonnet 4 and Gemini 2.5 Pro are much better at following the spirit of my system prompt rather than the letter. I do not use OpenAI models regularly, so I’m not sure about them.
BrawnyBadger53 · 1h ago
Similar experience, feels very ironic
airstrike · 1h ago
In my experience, whenever you do that, the model then overindexes on criticism and will nitpick even minor stuff. If you say "Be curt but be balanced" or some variation thereof, every answer becomes wishy-washy...
AznHisoka · 12m ago
Yeah, when I tell it to "Just be honest dude" it then tells me I'm dead wrong. I inevitably follow up with "No, not that KIND of honest!"
cheschire · 1h ago
Imagine what happens to Dutch culture when American trained AI tools force American cultural norms via the Dutch language onto the youngest generation.
And I’m not implying intent here. It’s simply a matter of source material quantity. Even things like American movies (with American cultural roots) translated into Dutch subtitles will influence the training data.
scott_w · 1h ago
Your comment reminds me of quirks of translations from Japanese to English where you see common phrases reused in the “wrong” context for English. “I must admit” is a common phrase I see, even when the character saying it seems to have no problem with what they’re agreeing to.
jstummbillig · 1h ago
What will happen? Californication has been around for a while, and, if anything, I would argue that AI is by design less biased than pop culture.
cheschire · 1h ago
Pop culture is not the intent of “study mode”.
siva7 · 1h ago
Let's face it. There is no one size fits all for this category. There won't be a single winner that takes it all. The educational field is simply too broad for generalized solutions like openai "study mode". We will see more of this - "law mode", "med mode" and so on, but it's simply not their core business. What are openai and co trying to achieve here? Continuing until FTC breaks them up?
wafflemaker · 57m ago
Reading the special prompt that makes the new mode, I discovered that in my prompting I never used enough ALL CAPS.
Is Trump, with his often ALL CAPS SENTENCES on to something? Is he training AI?
Need to check these bindings. Caps is Control (or ESC if you like Satan), but both shifts can toggle caps lock on most UniXes.
blueboo · 57m ago
Contrast the incentives with a real tutor and those expressed in the Study Mode prompt. Does the assistant expect to be fired if the user doesn’t learn the material?
herval · 7m ago
Most teachers are not at threat of being fired if individual kids don’t learn something. I’m not sure that’s such an important part of the incentive system…
I thought the AI safety risk stuff was very over-blown in the beginning. I'm kinda embarrassed to admit this: About 5/6 months ago, right when ChatGPT was in it's insane sycophancy mode I guess, I ended up locked in for a weekend with it...in...what was in retrospect, a kinda crazy place. I went into physics and the universe with it and got to the end thinking..."damn, did I invent some physics???" Every instinct as a person who understands how LLMs work was telling me this is crazy LLMbabble, but another part of me, sometimes even louder, was like "this is genuinely interesting stuff!" - and the LLM kept telling me it was genuinely interesting stuff and I should continue - I even emailed a friend a "wow look at this" email (he was like, dude, no...) I talked to my wife about it right after and she basically had me log off and go for a walk. I don't think I would have gotten into a thinking loop if my wife wasn't there, but maybe, and then that would have been bad. I feel kinda stupid admitting this, but I wanted to share because I do now wonder if this kinda stuff may end up being worse than we expect? Maybe I'm just particularly susceptible to flattery or have a mental illness?
"People Are Losing Loved Ones to AI-Fueled Spiritual Fantasies" https://news.ycombinator.com/item?id=43890649
Some people are also more susceptible to various too-good-to-be-true scams without alarm bells going off, or to hypnosis or cold reading or soothsayers etc. Or even propaganda radicalization rabbit holes via recommendation algorithms.
It's probably quite difficult and shameful-feeling for someone to admit that this happened to them, so they may insist it was different or something. It's also a warning sign when a user talks about "my chatgpt" as if it was a pet they grew and that the user has awakened it and now they together explore the universe and consciousness and then the user asks for a summary writeup and they try to send it to physicists or other experts and of course they are upset when they don't recognize the genius.
Unlike your regular scam, there's an element of "boiling frog" with LLMs.
It can start out reasonably, but very slowly over time it shifts. Unlike scammers looking for their payday, this is unlimited and it has all the time in the world to drag you in.
I've noticed it reworking in content of previous conversations from months ago. The scary thing is that's only when I've noticed it, I can only imagine how much it's tailoring everything for me in ways I don't notice.
Everyone really should be regularly clearing their past conversations and disable saving/training.
ChatGPT in its sycophancy era made me buy a $35 domain and waste a Saturday on a product which had no future. It hyped me up beyond reason for the idea of an online, worldwide, liability-only insurance for cruising sailboats, similar to SafetyWing. "Great, now you're thinking like a true entrepreneur!"
In retrospect, I fell for it because the onset of its sycophancy was immediate and without any additional signals like maybe a patch note from OpenAI.
- sycophancy tendency & susceptibility
- need for memory support when planning a large project
- when re-writing a document/prose, gen ai gives me an appreciation for my ability to collect facts, as the Gen AI gizmo refines the Composition and Structure
(Edit: Thanks to the couple people who emailed me, don't worry I'm laying off the LLM sauce these days :))
This seems uncannily similar to anti-COVID vaccination thinking. It isn't people being stupid because if you dig you can find heaps of papers and references and details and facts. So much so that the human mind can be easily convinced. Are those facts and details accurate? I doubt it, but the volume of slightly wrong source documents seems to add up to something convincing.
Also similar to how finance people made tranches of bad loans and packaged them into better rated debt, magically. It seems to make sense at each step but it is ultimately an illusion.
Nature is overwhelmingly non-linear. Most of human scientific progress is based on linear understandings.
Linear as in for this input you get this output. We've made astounding progress.
Its just not a complete understanding of the natural world because most of reality can't actually be modeled linearly.
https://gizmodo.com/billionaires-convince-themselves-ai-is-c...
Something which is very sorely missing from modern education is critical thinking. It's a phrase that's easy to gloss over without understanding the meaning. Being skilled at always including the aspect of "what could be wrong with this idea" and actually doing it in daily life isn't something that just automatically happens with everyone. Education tends to be the instructor, book, and facts are just correct and you should memorize this and be able to repeat it later. Instead of here are 4 slightly or not so slightly different takes on the same subject followed by analyzing and evaluating each compared to the others.
If you're just some guy who maybe likes reading popular science books and you've come to suspect that you've made a physics breakthrough with the help of an LLM, there are a dozen questions that you should automatically have in your mind to temper your enthusiasm. It is, of course, not impossible that a physics breakthrough could start with some guy having an idea, but in no, actually literally 0, circumstances could an amateur be certain that this was true over a weekend chatting with an LLM. You should know that it takes a lot of work to be sure or even excited about that kind of thing. You should have a solid knowledge of what you don't know.
It’s a huuuuuuuuuuuuge logical leap from LLM conversation yo novel physics. So huge a leap anyone ought to be immediately suspicious.
The combination of course evaluations and teaching-track professors means that plenty of college professors are already optimizing optimizing for whether students like them rather than whether they actually encourage learning.
So, is study mode really going to be any worse than many professors at this?
But it doesn’t work much …
(I’m serious, these things are so weird that it would probably work.)
"Here's your brutally honest answer–just the hard truth, no fluff: [...]"
I don't know whether that's better or worse than the fake flattery.
And I’m not implying intent here. It’s simply a matter of source material quantity. Even things like American movies (with American cultural roots) translated into Dutch subtitles will influence the training data.
Is Trump, with his often ALL CAPS SENTENCES on to something? Is he training AI?
Need to check these bindings. Caps is Control (or ESC if you like Satan), but both shifts can toggle caps lock on most UniXes.