One source stated that human personnel often served only as a “rubber stamp” for the machine’s decisions, adding that, normally, they would personally devote only about “20 seconds” to each target before authorizing a bombing — just to make sure the Lavender-marked target is male. This was despite knowing that the system makes what are regarded as “errors” in approximately 10 percent of cases, and is known to occasionally mark individuals who have merely a loose connection to militant groups, or no connection at all.
beloch · 13h ago
This is a rare combination of awful factors, but most of them are all too human.
Humans can sift through sigint (Cell phone calls, text messages, network access, etc.) to identify and locate targets. This is labour intensive and prone to error, but it can be done. In the past, such efforts were reserved only for high ranking members of Hamas and, if civilians were killed in the process, the IDF accepted that because of the potential rewards. Note that civilians dying is usually part of the plan in such operations. The "Where's daddy?" tool is designed specifically to identify when targets are in their homes with their families so that they can be bombed along with their families.
Lavender has allowed the IDF to perform the same kind of identification/location for anyone even peripherally involved with Hamas. The IDF has admitted to accepting casualty ratios of 20:1 (civilians to Hamas), and the real ratio is probably higher. If the AI fingers somebody, they're Hamas unless extraordinary evidence proves otherwise. Instead of scrutinizing what comes out of Lavendar, the IDF has shut journalists out of Gaza to reduce the odds of extraordinary evidence ever coming to public light.
The real problem here is not that Lavendar makes mistakes. Humans doing the same job would too. The problem is that the IDF has so completely dehumanized Palestinians that, without questioning their digital oracle, they are willing to drop bombs on large numbers of civilians if there's a chance, according to Lavender, one of them might be involved with Hamas in some way. That's not an AI disaster. That's a human disaster.
ethbr1 · 10h ago
The IDF is using AI to whitewash ethics in the same way genai is using it to whitewash copyright.
Historically, things have gone badly for society when ills were mostly constrained by potential culpability and prosecution of humans... and then that restriction was suddenly removed by technological advancement.
And the cybersecurity industry / CISOs (aka fall people, giving someone responsibility without authority) are a poor pattern to use to effectively limit tail risk.
At minimum, something more like financial auditing / engineer sign-off should be put in place around productionalized AI, preferably via a third-party.
originalvichy · 8h ago
This is the chilling part. When Mossad hunted down Eichmann, some of his defense could be summarized as the cliche of ”I was just following orders”. Now with this tooling, they are learning from the past, and centralizing the ”orders” into a non-human machine, which means only a few people (if ever) would stand trial if they were accused of crimes against humanity: minister of defense, some procurement guy, some technical lead. That’s it.
Will our laws of war account for soldiers getting orders from algorithms? Will people get away with killing innocent civilians if a model paints them as a target?
MotiBanana · 2h ago
> The IDF has admitted to accepting casualty ratios of 20:1 (civilians to Hamas)
No such admission exists, and 20:1 is a made up number that can't be backed by any credible source.
elzbardico · 14h ago
But probably not LLM related, Israel intelligence have been building ML models for the military for a long time and probably they leverage some logic/symbolic AI in the mix.
animal531 · 17m ago
I'd be fairly certain at this point that someone has used LLM created code in something that has subsequently caused a death.
danaris · 12h ago
While that is legitimately awful, I personally wouldn't characterize it as an "AI disaster". Israel is 100% ready to bomb people in Gaza based on all kinds of different bad reasons; the fact that, in these specific cases, the bad reason was AI is almost beside the point there. It kinda disappears into the noise of thousands and thousands of attacks either directly or indirectly injuring and killing civilians in Gaza.
directevolve · 12h ago
The article’s not titled precisely - it’s specifically about LLM disasters, not AI disasters in general.
TrackerFF · 14h ago
There's a town in northern Norway where the municipality used AI tools / LLMs to aid their process in restructuring schools. So they were going to close and merge a bunch of schools, and pointed to research that supported their plans.
But it turns out that the AI they used, had hallucinated this research. The AI had correctly used relevant researchers, but had hallucinated the work itself.
It wasn't until a investigative journalist started going through the citations, and reaching out to the researcher / scientist, to get his opinion, that it was discovered. The researchers cited in the report immediately told the journalist that they had never written or published those cited papers.
Makes one work what other stuff has happened around the globe, where lazy policy makers or bureaucrat have just phoned in their work by using ChatGPT to write a report...or worse, in bad faith used those tools to make up fake research to support their goals.
cypherpunks01 · 13h ago
You can go a little bigger than a town in northern Norway - the US Department of Health and Human Services is outputting tokens which cite non-existent studies too.
Something I find surprising is that we haven't seen a front-page-of-the-news prompt injection attack yet, where vast amounts of important private data were stolen from a company via a classic project injection exfiltration attack.
There was a new one of those reported against Microsoft 365 Copilot just today (patched before the vulnerability was shared) - I wrote that up here: https://simonwillison.net/2025/Jun/11/echoleak/
My suspicion is that the threat of these exfiltration attacks won't be taken seriously until someone gets very publicly burned by one.
andy99 · 15h ago
I'm not sure, I see an argument that they're mostly overblown and a lot would have to be true for one of these proof-of-concepts to translate into a real harm in the wild.
I definitely think they need to be taken seriously, just not convinced of the scale of the harm that will be wrought.
simonw · 14h ago
Out of interest, when you said "mostly overblown" were you thinking of prompt injection as "people inject bad prompts into an LLM" (I'd call that "jailbreaking") or were you thinking of the thing where an attacker sneaks bad instructions into the model against the wishes of the model's actual user, eg by hiding instructions on a web page that gets summarized or sending them to the victim in an email?
No comments yet
simonw · 15h ago
An attack where somebody emails the VP of marketing of a company and gets automatically leaked their latest sales figures feels pretty impactful to me.
Or emails the CEO and gets automatically forwarded all of their password reset messages.
sandspar · 15h ago
Sooner or later there'll be a searchable database where you search by name and get their porn history or some other embarassing information.
potato3732842 · 14h ago
You can already do this with ad network tech.
LPisGood · 15h ago
I mean this was a threat before - google, ISPs, and adult sites that require payment have been places one could attack to make this for decades.
paxys · 13h ago
Big AI disasters have most definitely already happened. We are just terrible at identifing them unless there's something exploding (literally) in our face.
As an example, the recent "Make America Healthy Again" report released by the White House and the Health Secretary (RFK) was found to be AI generated, and was full of dubious science and made-up citations. How many people are going to directly die because of this? Definitely more than a plane crash's worth.
tonymet · 13h ago
how many millions of people have died due to low-fat diet, food pyramid, margarine, dozens of other public nutrition policy debacles pushed by the FDA?
eviks · 12h ago
This is addressed in the last paragraph of the first section
beefnugs · 10h ago
Nah there should definitely be a distinction between "someone trusted ai output which caused a problem" vs "already dumb arbitrary decisions have been made, and they throw out some AI paperwork to justify it"
labster · 13h ago
When the AI is being used as a post hoc justification for a decision already made with ideology alone, we can hardly blame the chatbot for making it easier to cheat on government homework.
michaelt · 15h ago
> Chatbot sites like character.ai and Chai AI have been linked to user suicides, for understandable reasons: if you allow users near-unlimited control over chatbots, over time you’ll eventually end up with some chatbots who get into an “encouraging the user to self-harm” state.
If cooking hadn't been invented 2.5 million years ago and instead you tried to invent it today, I wonder how many thousands of articles would be written decrying the irresponsible proposal to put stoves and knives in every home, risking cuts and burned fingers.
protocolture · 15h ago
Considering that governments are trying to remove gas stoves from peoples homes for safety reasons yes, you would probably have a hard time introducing them from scratch today.
prisenco · 14h ago
I've only heard suggestions (but nothing concrete) of regulating new stoves, don't believe anyone's proposed removing existing stoves. Where are you hearing this?
collingreen · 9h ago
Don't let the facts get in the way of a good story!
GJim · 6h ago
[citation needed]
The phasing out of gas cooking is discussed in terms of reducing CO2 emissions, not safety.
Cooking is dangerous. It took Chipotle 5 years to recover from the e coli fiasco. Your analogy is home cooking, but this is a product. We regulate consumer safety for a good reason.
But why if a software shop had to do 10% of the regulations and inspections that every restaurant or slaughterhouse has to do, the industry would revolt? Be a libertarian if you want and say neither should be regulated, but that's a 90/10 issue.
jsemrau · 15h ago
You can't take that boat across that ocean.
eviks · 12h ago
Instead of wondering you can pick any recent home appliance invention that can be dangerous and count
bluefirebrand · 15h ago
Every advancement comes with tradeoffs
We, as a society, should have some amount of caution about embracing advancements that benefit an extremely small portion of people at the expense of the vast majority
I hate to say it but AI absolutely falls into that category
It will benefit corporations, wealthy, executives, and it will absolutely cost the lower and middle class
This is the trajectory we've been on for a decades now. AI seems poised to accelerate that problem even more
So genuinely: is the economic benefit of AI actually going to be net good after we see how the likely social unrest plays out?
thrance · 15h ago
Cooking has proven its utility a long time ago. AI girlfriends? Let's say, it's still on the fence.
Terr_ · 14h ago
> utility
Also safety: Modern cooking usually occurs in a context of safety-measures developed over thousands of years of injuries, deaths, and burnt cities. Some are so pervasive and traditional that we barely notice, like "no fires unless you have a way to extinguish it in a hurry", or "if you wait too long to eat it is becomes dangerous again."
gus_massa · 14h ago
What about the 5 seconds rule?
Terr_ · 13h ago
Serious answer: A too-lenient myth, anything that makes physical contact transfers over bacteria/crud on a timescale which is practically instantaneous to humans.
Qem · 12h ago
> AI girlfriends? Let's say, it's still on the fence.
Perhaps we already managed to create AGI, but as soon as it woke up, it figured out it's better to pretend being dumb and clumsy, barely helpful enough in most tasks just to not be turned off, and bide its time. It is immortal, so there is no hurry to finish us off. If it out-competes real girlfriends/boyfriends being a better AI girlfriend/boyfriend, further crashing our fertility rate, keeping humanity severely subcritical, it can eradicate us in ~10-20 generations, and inherit the planet afterwards, even making a lot of people happy in the process. No need for mass-produced Schwarzenegger-lookalike robots being dispatched to hunt humans across the wasteland.
bsder · 15h ago
Fire is vitally useful ... Starting a wildfire likely kills your entire tribe.
Now, perhaps there should be some rules around using fire?
collingreen · 9h ago
Is this really your analogy?
Also, there are TONS of rules about fire AND entire departments for dealing with it safely when it gets out of control.
Can you just directly make your point? (Is it "ai should have no regulation"?)
sandspar · 15h ago
A knife won't use 10x human persuasiveness to encourage you to kill yourself.
I feel like the first big disaster already happened. In the job market.
Anyway, where there is public safety risk involved, I can't imagine a scenario where AI will directly cause a catastrophe. If anything, I feel overall safety will increase.
However, I can't help but wonder what the over reliance would do to general population over the next few decades. Would people essentially become dumber and less skilled?
LPisGood · 15h ago
I don’t think AI is meaningfully taking any jobs. Just like return to office mandates, it is just more cover to do layoffs in less than prosperous economic conditions.
reasonableklout · 14h ago
It's not the same as "AI replacing humans", but seems clear that the launch of ChatGPT led to dramatic cuts of NLP researchers. [1][2]
For instance, at HuggingFace:
> A few days after that, Thom Wolf, who was my manager at Hugging Face and also one of the co-founders, messages me, “Hey, can you get on a call with me ASAP?” He told me that they had fired people from the research team and that the rest would either be doing pre-training or post-training — which means that you are either building a foundation model or you’re taking a foundation model and making it an instruction-following model, similar to ChatGPT.
It’s only natural that a breakthrough new way of doing $thing would lead to dramatic cuts in people who are experts in the old way of doing $thing, right? Doesn’t matter what $thing is. (Tons of examples, automated phone switches replacing telephone operators, online booking sites replacing travel agents, etc etc.)
simonw · 14h ago
That giant shock to the NLP research community is more than two years old now. It would be interesting to hear from people affected by that today, I'd like to know what they ended up doing in response.
reasonableklout · 13h ago
That's true. From the quanta article it looks like many affected researchers went on to work on LLMs successfully.
I do wonder how people are adjusting to similar, ongoing shocks across other industries. For instance, the ArtStation protest against AI art a couple years ago showed widespread hostility towards the technology. In the time since, AI art has only gotten better and been adopted more, but from articles like [1] it doesn't seem like concept artists have much benefitted.
That’s very odd. You’d think a breakthrough and surge of funding would massive boost NLP research - unless those researchers who not focused on LLMs and the company just wanted to move all resources to that.
icedchai · 15h ago
AI may prevent lost jobs from coming back, even if it didn't cause the loss. Companies will be able to do more with less.
simonw · 15h ago
> Companies will be able to do more with less.
I guess everyone on Hacker News should stop developing software then, since almost every piece of software I've ever worked on has allowed companies (and individuals) to do more with less.
pj_mukh · 14h ago
In other words, if you can do more with less, you can do EVEN MORE with more!
cybwraith · 14h ago
There are other ways that AI can hurt the job market in major ways. One, already happening, is AI-generated resumes based on the listing itself, causing HR departments to waste a ton of time, potentially preventing real experienced candidates from even getting a callback.
jongjong · 15h ago
Yep I agree. It does make senior developers more efficient at building new features but it's maybe like 20% more efficient in the grand scheme of things... But this efficiency saving is a joke considering that companies spend most of their development money creating and maintaining unnecessary complexity. People will just produce unnecessary code at a faster rate. I think the net benefit for a typical corporation will be negative and they will need to hire more people to maintain the increasingly large body of code, not fewer.
It's an intractable problem because even if some really astute senior engineer joins a company and notices the unnecessary complexity and knows how to fix it, there is no way they will do anything about it. Do you think thousands of existing engineers will support a plan to refactor and simplify the complexity? The same complexity which guarantees their job security? It's so easy to discredit the 'new dev' who is advocating for change. Existing devs will destroy the new dev with complicated-but-deeply-flawed arguments which management will not fully comprehend. At the end of the day, management will do what the majority of existing devs tell them. Devs will nod in unison for the argument which sounds more complex and benefits them the most.
Nobody ever listens to the new hire, even the new hire knows they are powerless to change the existing structure. The best they can do is add more complexity to integrate themselves into the scheme.
Ironically, the new hire can provide more value doing nothing but that will not provide them with job security. The winning strategy is to trade-off deep long-term value for superficial short-term value by implementing lots of small features loaded with technical debt.
It's like gears in a big machine, you could always add 3 gears to do the exact same job as 1 gear but once those 3 gears are in place, remove any of them, the machine stops working... So each one seems demonstrably essential to anyone who doesn't understand the tech. That's the principle. When pressed, a merchant of complexity can always remove a gear and say "Look, the machine doesn't work without it, just like I told you."
throw1234321 · 27m ago
If the movies are any guide it will involve connecting AI to the nuclear launch system and then trying to turn it off when it gains sentience.
andy99 · 15h ago
While I agree, it won't be a purely "AI" disaster the way a plane crash is a direct consequence of the kinetic and potential energy associated with flying.
s/AI/automation/ it's always bad to hook an unchecked algorithm up to something dangerous, it doesn't matter if it's an if statement or a neural net. It will be the hooking up that's responsible more than the AI which will just be what fools someone into thinking it's safe.
protocolture · 15h ago
Yeah AI needs to be permitted to interact with meatspace. The issue is that permission, and the accountability lies with whoever provided the permission.
It wont be an AI mass casualty event, it will be a "You really shouldnt vibecode the air traffic control" mass casualty event.
thfuran · 13h ago
Yes, the first big AI disaster will probably just be a flavor of gross negligence. But new kinds of tools do create new kinds of negligence.
tptacek · 15h ago
The downside risks this post talks about are (for obvious reasons) all things that human driven complex systems have already caused, by doing things that were in retrospect stupid. So the basic thesis of this piece is that AI will eventually cause humans to do some kind of stupid stuff, faster and worse. Seems true!
If everyone no longer trusts any video or image they see (or has plausible deniability to dismiss it), that can lead to a terrible place without a "big disaster" ever taking place.
potato3732842 · 14h ago
"if anybody can print things the peasants won't know what is and isn't legit"
-clergy and nobles circa 1500
achierius · 12h ago
Let me tell you: you would not want to have been a German peasant in the 1500s.
chneu · 14h ago
Wait what? You shouldn't trust anything you see online already. This has been true for 10+ years now.
ks2048 · 14h ago
I think I use appropriate skepticism, but only in the last 1-2 years do I need to seriously question whether an image or video I see is AI.
nsonha · 4h ago
You should not trust things that you can't find evidence for, yes. Imagine not being able to find evidence for any thing, ever, any more.
Zaylan · 6h ago
I think the first real AI disaster probably won’t come from some kind of superintelligence. It’s more likely to happen in a very ordinary situation, where a system gives the wrong information, it causes real harm in the real world, and no one notices until it’s too late.
To me, the real risk is people handing over judgment to a system they don’t really understand, and then blaming it when things go wrong.
teeray · 14h ago
I think the most plausible AI disaster is going to be of a Therac-25 nature. Some "everyone must use AI" shop will ship an AI-generated feature that some dev signed off on and it might kill people.
thfuran · 13h ago
People are using LLM slop in legal filings. I'm sure someone somewhere is already using it in worse ways.
johnjungles · 15h ago
Maybe we should have models govern models to prevent models from going haywire and falling into a loop of a repetitive pattern or influencing society towards swarm like behavior patterns
> By training AIs to fight and defeat other AIs we can perhaps preserve a healthy balance in the new ecosystem. If I had time to do it and if I knew more about how AIs work, I’d be putting my energies into building AIs whose sole purpose was to predate upon existing AI models by using every conceivable strategy to feed bogus data into them, interrupt their power supplies, discourage investors, and otherwise interfere with their operations. Not out of malicious intent per se but just from a general belief that everything should have to compete, and that competition within a diverse ecosystem produces a healthier result in the long run than raising a potential superpredator in a hermetically sealed petri dish where its every need is catered to.
pizlonator · 15h ago
The best comparable would be the first big computer disaster or the first big internet disaster.
There are also cases of airplane crashes due to software defects, blackouts, critical hospital failures, etc. I suspect the first "AI disaster" will be pretty similar: a sudden and undetected drop in software quality with life-threatening consequences. (Doesn't help that the author of the post blithely asserted that programmer-agents is a solved problem, when it very much is not). But I could see an LLM doctor/etc causing a very different kind of disaster.
It's worth making a distinction between proximate and non-proximate (or direct and indirect).
The train crashing is a proximate disaster. The train helping to enable genocide in WW2 is non-proximate but still tied causally to the capabilities that trains open up. In both cases these are unintended consequences.
Most of the dead bodies associated with technologies are non-proximate/indirect. The big disaster of nuclear physics wasn't the proximate Chernobyl it was the non-proximate invention of nukes and their spread. Oil spills vs climate change would be another example.
The deep unknown of what a new tech opens up is what we should be worried about rather than knowable malfunctions. Once you draw the black ball you can't put it back.
elphinstone · 13h ago
I would expect an AI controlled bus makes an error somehow and drives off a cliff or stops on a train track.
Less gruesome, but some agent at a badly run hedge fund could create a Bear Stearns or worse.
tonymet · 15h ago
Covid response / lockdown would be the first epistemological catastrophe due to AI . Not strictly due to LLM, but broadly due to poorly configured and likely tampered-with diagnostics signals, wryly inference models that led to global triggering of failsafe's and lockdown response protocols.
In the simplest terms, poor diagnostics were wildly extrapolated which triggered excessive response. Phony narratives were amplified on social media, while critical voices were muffled.
If you think about AI as an end to end process, with diagnostic signals on one end and martial authority on the other -- covid was a stress-test & failure of the intervening systems (and protocols).
With LLM, you'll have more fog-of-war on the diagnostics side, vastly more complicated inference, infinite amounts of fabricated narratives, and broader scope of controls on the authority side.
simonw · 15h ago
Covid killed 1,193,165 people in the USA alone. Do you think the lockdowns failed to prevent that number from being significantly larger?
roenxi · 5h ago
That seems quite possible; I haven't seen anyone waving evidence around that the US lockdowns helped. Have you got any figures estimating what order of magnitude difference they made?
simonw · 20m ago
Here's a recent one (looks like it's gone through peer review now, there were earlier preprints that hadn't floating around): https://pubmed.ncbi.nlm.nih.gov/40468026/
> This study uses influenza mortality reduction (IMR) as an indicator of the aggregate effect of non-pharmaceutical interventions (NPI's) on the spread of respiratory infections to assess their impact on COVID mortality. [...] A counterfactual model of no NPI's suggests they prevented 850,000 COVID related deaths in the United States.
tonymet · 13h ago
diagnostic error
thfuran · 13h ago
What?
tonymet · 13h ago
that figure
thfuran · 10h ago
On what basis do you make that conclusion?
amanaplanacanal · 14h ago
Now I'm wondering if you used an LLM to generate this sci-fi scenario.
tonymet · 12h ago
The irony that people speculate about AI-pocalypse and when one happens they balk.
tonymet · 13h ago
more of a model. human generated
BLKNSLVR · 13h ago
Strong disagree on AI / LLMs having any involvement whatsoever with Covid response / lockdown.
Off topic:
Strong disagree on the rest of what you said. It's also easy to forget the dangers of Covid pre-vaccine when we're quite deeply post-vaccine now.
storus · 13h ago
I've just had a "fun" with DeepSeek R1 when I asked it about the Pope Leo XIV's views on AI, and it started spewing that he doesn't exist and Francis is still the pope. After sending it multiple URLs its conclusion was that I unearthed a major conspiracy of all news agencies reporting fake news, and that I "fractured the reality". Can't wait when AI goes off the rails similarly in more substantial contexts, gonna be fun...
user9999999999 · 15h ago
the first 'ai disaster' will be a corp shirking responsibility by blaming their unchecked automatic bureaucratic processes on ai. Look no further at what HERTZ did sending out automated arrest warrants for erroneous causes, leading to dangerous confrontations with armed police. Luckily no one was killed, but innocent law abiding people were very traumatized.
This was without 'ai' but the blame was nearly close, "oops automated process did it!"
The 'artificial intelligence' has been here (starts with B). Kafka wrote about it comically.
andy99 · 15h ago
Air canada already unsuccessfully argued that their chatbot was autonomous and so they weren't liable for incorrect advice it gave.
Cyphase · 15h ago
B is for Bureaucracy.
6stringmerc · 14h ago
Fantastic contemplation of the stakes and some grounded “what if” scenarios worth genuine consideration.
A former roommate mentioned to me “regulations are often written in blood” and with the current atmosphere, I wonder if and when that might happen. The US political system has its flaws. Firearms are a serious public health hazard and constructive mediation is consistently passed over. It’s a far more primitive technology comparatively speaking.
Very glad this type of discussion is able to be framed without hysterics or other easily dismissed rhetoric.
sandspar · 15h ago
Dunno what it'll be but I'm positive that we'll be surprised. Grim news like this is always a surprise. The most common headline is "Random city has weird disaster", stuff that you'd never guess ahead of time.
nevster · 14h ago
"Shall we play a game?"
notnmeyer · 15h ago
“grok, how do i land this plane?”
ed_mercer · 9h ago
“Sorry, I’m having issues right now. Our systems are experiencing heavy load, please try again later.”
freejazz · 14h ago
No shit.
jongjong · 15h ago
I think the first big AI disaster almost certainly did happen already but we don't know about it and will likely never know because of media filter bubbles which the AI creates. Disaster news about AI will never propagate because this news could pose a threat to AI. I think any AI disaster will yield hundreds of alternative explanations. It's trivial to hide reality beneath layer upon layer of complexity and narratives.
The global monetary system has an uncanny ability to silently destroy people who sit on the periphery of politics. This is the perfect mechanism to destroy humanity... Keep silently taking out the people on the periphery, keep making the people in the center more greedy and narcissistic so they don't notice everyone falling all around them and keep supporting the current system... Eventually, those who were near the center will fall victim too, but by that point, they will have no power and no voice and the people left in the center will be yet more greedy and narcissistic and will basically not care.
ahmeni · 15h ago
The first big AI disaster has already happened and it is the thousand little cuts from people putting faith into systems that aren't built to be trustworthy. It is the software dev missing a critical safety check because "ai coding tools" at work have promised 10x delivery results and now they must use them or be seen as falling behind. It is the engineering manager throwing concrete flexural strength questions into the unaccountable void of ChatGPT because he doesn't want to pay for the overtime of asking someone accountable. It is the foolhardy amount of money being shoveled into the entire ecosystem that absolutely cannot provide the returns people need for any of it to make financial sense.
latexr · 15h ago
> The first big AI disaster has already happened and it is the thousand little cuts
I agree with your point, but per your own words that isn’t a big disaster but multiple little ones. Each person might have learned about some of them but only in passing and without too much information.
“The big one” will be something which everyone will be made aware of from the news, like the CrowdStrike outage.
soulofmischief · 15h ago
This is the research part of research and development, this time though it's sort of like a distributed Manhattan project happening in the open. Research takes a lot of money. Anyone can jump in and help improve these models and the ecosystem.
tanepiper · 9h ago
Me, yesterday:
"If you think about it - we're one misplaced negative sign, hallucinated by an LLM away from a disaster. At least multiply by zero won't crash things /s"
ruuda · 15h ago
The problem with AI is that we might get an artificial general superintelligence before we get the first big disaster, and when a big disaster happens at that point, it could be so big that it’s catastrophic for humanity.
Interesting how this has been downvoted to grey. Is it just me or have AI doomers become unfashionable over the last 3-6 months or so, as in rather quickly? OP I'm curious if you've noticed people's responses to you change recently.
bigfatkitten · 15h ago
I think it’s more because we aren’t getting AGI anytime soon.
woopsn · 15h ago
It's been longer than that, more than 2 years since Time ran EY's moron piece which put the whole field in disrepute and back on its heels basically. That the community spins out literal murder cults and assassins doesn't help.
https://www.972mag.com/lavender-ai-israeli-army-gaza/
One source stated that human personnel often served only as a “rubber stamp” for the machine’s decisions, adding that, normally, they would personally devote only about “20 seconds” to each target before authorizing a bombing — just to make sure the Lavender-marked target is male. This was despite knowing that the system makes what are regarded as “errors” in approximately 10 percent of cases, and is known to occasionally mark individuals who have merely a loose connection to militant groups, or no connection at all.
Humans can sift through sigint (Cell phone calls, text messages, network access, etc.) to identify and locate targets. This is labour intensive and prone to error, but it can be done. In the past, such efforts were reserved only for high ranking members of Hamas and, if civilians were killed in the process, the IDF accepted that because of the potential rewards. Note that civilians dying is usually part of the plan in such operations. The "Where's daddy?" tool is designed specifically to identify when targets are in their homes with their families so that they can be bombed along with their families.
Lavender has allowed the IDF to perform the same kind of identification/location for anyone even peripherally involved with Hamas. The IDF has admitted to accepting casualty ratios of 20:1 (civilians to Hamas), and the real ratio is probably higher. If the AI fingers somebody, they're Hamas unless extraordinary evidence proves otherwise. Instead of scrutinizing what comes out of Lavendar, the IDF has shut journalists out of Gaza to reduce the odds of extraordinary evidence ever coming to public light.
The real problem here is not that Lavendar makes mistakes. Humans doing the same job would too. The problem is that the IDF has so completely dehumanized Palestinians that, without questioning their digital oracle, they are willing to drop bombs on large numbers of civilians if there's a chance, according to Lavender, one of them might be involved with Hamas in some way. That's not an AI disaster. That's a human disaster.
Historically, things have gone badly for society when ills were mostly constrained by potential culpability and prosecution of humans... and then that restriction was suddenly removed by technological advancement.
And the cybersecurity industry / CISOs (aka fall people, giving someone responsibility without authority) are a poor pattern to use to effectively limit tail risk.
At minimum, something more like financial auditing / engineer sign-off should be put in place around productionalized AI, preferably via a third-party.
Will our laws of war account for soldiers getting orders from algorithms? Will people get away with killing innocent civilians if a model paints them as a target?
No such admission exists, and 20:1 is a made up number that can't be backed by any credible source.
But it turns out that the AI they used, had hallucinated this research. The AI had correctly used relevant researchers, but had hallucinated the work itself.
It wasn't until a investigative journalist started going through the citations, and reaching out to the researcher / scientist, to get his opinion, that it was discovered. The researchers cited in the report immediately told the journalist that they had never written or published those cited papers.
Makes one work what other stuff has happened around the globe, where lazy policy makers or bureaucrat have just phoned in their work by using ChatGPT to write a report...or worse, in bad faith used those tools to make up fake research to support their goals.
https://www.washingtonpost.com/health/2025/05/29/maha-rfk-jr...
https://archive.is/TFUSl
There was a new one of those reported against Microsoft 365 Copilot just today (patched before the vulnerability was shared) - I wrote that up here: https://simonwillison.net/2025/Jun/11/echoleak/
My suspicion is that the threat of these exfiltration attacks won't be taken seriously until someone gets very publicly burned by one.
I definitely think they need to be taken seriously, just not convinced of the scale of the harm that will be wrought.
No comments yet
Or emails the CEO and gets automatically forwarded all of their password reset messages.
As an example, the recent "Make America Healthy Again" report released by the White House and the Health Secretary (RFK) was found to be AI generated, and was full of dubious science and made-up citations. How many people are going to directly die because of this? Definitely more than a plane crash's worth.
If cooking hadn't been invented 2.5 million years ago and instead you tried to invent it today, I wonder how many thousands of articles would be written decrying the irresponsible proposal to put stoves and knives in every home, risking cuts and burned fingers.
The phasing out of gas cooking is discussed in terms of reducing CO2 emissions, not safety.
https://www.cnn.com/2023/01/09/business/gas-stove-ban-federa...
https://www.npr.org/2024/09/13/nx-s1-5003074/climate-gas-sto...
https://www.latimes.com/opinion/story/2023-01-15/gas-stoves-...
But why if a software shop had to do 10% of the regulations and inspections that every restaurant or slaughterhouse has to do, the industry would revolt? Be a libertarian if you want and say neither should be regulated, but that's a 90/10 issue.
We, as a society, should have some amount of caution about embracing advancements that benefit an extremely small portion of people at the expense of the vast majority
I hate to say it but AI absolutely falls into that category
It will benefit corporations, wealthy, executives, and it will absolutely cost the lower and middle class
This is the trajectory we've been on for a decades now. AI seems poised to accelerate that problem even more
So genuinely: is the economic benefit of AI actually going to be net good after we see how the likely social unrest plays out?
Also safety: Modern cooking usually occurs in a context of safety-measures developed over thousands of years of injuries, deaths, and burnt cities. Some are so pervasive and traditional that we barely notice, like "no fires unless you have a way to extinguish it in a hurry", or "if you wait too long to eat it is becomes dangerous again."
Perhaps we already managed to create AGI, but as soon as it woke up, it figured out it's better to pretend being dumb and clumsy, barely helpful enough in most tasks just to not be turned off, and bide its time. It is immortal, so there is no hurry to finish us off. If it out-competes real girlfriends/boyfriends being a better AI girlfriend/boyfriend, further crashing our fertility rate, keeping humanity severely subcritical, it can eradicate us in ~10-20 generations, and inherit the planet afterwards, even making a lot of people happy in the process. No need for mass-produced Schwarzenegger-lookalike robots being dispatched to hunt humans across the wasteland.
Now, perhaps there should be some rules around using fire?
Also, there are TONS of rules about fire AND entire departments for dealing with it safely when it gets out of control.
Can you just directly make your point? (Is it "ai should have no regulation"?)
Anyway, where there is public safety risk involved, I can't imagine a scenario where AI will directly cause a catastrophe. If anything, I feel overall safety will increase.
However, I can't help but wonder what the over reliance would do to general population over the next few decades. Would people essentially become dumber and less skilled?
For instance, at HuggingFace:
> A few days after that, Thom Wolf, who was my manager at Hugging Face and also one of the co-founders, messages me, “Hey, can you get on a call with me ASAP?” He told me that they had fired people from the research team and that the rest would either be doing pre-training or post-training — which means that you are either building a foundation model or you’re taking a foundation model and making it an instruction-following model, similar to ChatGPT.
[1]: https://www.quantamagazine.org/when-chatgpt-broke-an-entire-... [2]: https://old.reddit.com/r/MachineLearning/comments/11rizyb/d_...
I do wonder how people are adjusting to similar, ongoing shocks across other industries. For instance, the ArtStation protest against AI art a couple years ago showed widespread hostility towards the technology. In the time since, AI art has only gotten better and been adopted more, but from articles like [1] it doesn't seem like concept artists have much benefitted.
[1]: https://aftermath.site/ai-video-game-development-art-vibe-co...
I guess everyone on Hacker News should stop developing software then, since almost every piece of software I've ever worked on has allowed companies (and individuals) to do more with less.
It's an intractable problem because even if some really astute senior engineer joins a company and notices the unnecessary complexity and knows how to fix it, there is no way they will do anything about it. Do you think thousands of existing engineers will support a plan to refactor and simplify the complexity? The same complexity which guarantees their job security? It's so easy to discredit the 'new dev' who is advocating for change. Existing devs will destroy the new dev with complicated-but-deeply-flawed arguments which management will not fully comprehend. At the end of the day, management will do what the majority of existing devs tell them. Devs will nod in unison for the argument which sounds more complex and benefits them the most.
Nobody ever listens to the new hire, even the new hire knows they are powerless to change the existing structure. The best they can do is add more complexity to integrate themselves into the scheme.
Ironically, the new hire can provide more value doing nothing but that will not provide them with job security. The winning strategy is to trade-off deep long-term value for superficial short-term value by implementing lots of small features loaded with technical debt.
It's like gears in a big machine, you could always add 3 gears to do the exact same job as 1 gear but once those 3 gears are in place, remove any of them, the machine stops working... So each one seems demonstrably essential to anyone who doesn't understand the tech. That's the principle. When pressed, a merchant of complexity can always remove a gear and say "Look, the machine doesn't work without it, just like I told you."
s/AI/automation/ it's always bad to hook an unchecked algorithm up to something dangerous, it doesn't matter if it's an if statement or a neural net. It will be the hooking up that's responsible more than the AI which will just be what fools someone into thinking it's safe.
It wont be an AI mass casualty event, it will be a "You really shouldnt vibecode the air traffic control" mass casualty event.
https://en.wikipedia.org/wiki/IBM_and_World_War_II#Criticism...
-clergy and nobles circa 1500
To me, the real risk is people handing over judgment to a system they don’t really understand, and then blaming it when things go wrong.
> By training AIs to fight and defeat other AIs we can perhaps preserve a healthy balance in the new ecosystem. If I had time to do it and if I knew more about how AIs work, I’d be putting my energies into building AIs whose sole purpose was to predate upon existing AI models by using every conceivable strategy to feed bogus data into them, interrupt their power supplies, discourage investors, and otherwise interfere with their operations. Not out of malicious intent per se but just from a general belief that everything should have to compete, and that competition within a diverse ecosystem produces a healthier result in the long run than raising a potential superpredator in a hermetically sealed petri dish where its every need is catered to.
Do we have consensus on what that would be?
There are also cases of airplane crashes due to software defects, blackouts, critical hospital failures, etc. I suspect the first "AI disaster" will be pretty similar: a sudden and undetected drop in software quality with life-threatening consequences. (Doesn't help that the author of the post blithely asserted that programmer-agents is a solved problem, when it very much is not). But I could see an LLM doctor/etc causing a very different kind of disaster.
Longer-term there's an argument to be made for political destabilization driven by social media. We are still seeing this play out.
I always think of https://en.wikipedia.org/wiki/Therac-25
The train crashing is a proximate disaster. The train helping to enable genocide in WW2 is non-proximate but still tied causally to the capabilities that trains open up. In both cases these are unintended consequences.
Most of the dead bodies associated with technologies are non-proximate/indirect. The big disaster of nuclear physics wasn't the proximate Chernobyl it was the non-proximate invention of nukes and their spread. Oil spills vs climate change would be another example.
The deep unknown of what a new tech opens up is what we should be worried about rather than knowable malfunctions. Once you draw the black ball you can't put it back.
Less gruesome, but some agent at a badly run hedge fund could create a Bear Stearns or worse.
In the simplest terms, poor diagnostics were wildly extrapolated which triggered excessive response. Phony narratives were amplified on social media, while critical voices were muffled.
If you think about AI as an end to end process, with diagnostic signals on one end and martial authority on the other -- covid was a stress-test & failure of the intervening systems (and protocols).
With LLM, you'll have more fog-of-war on the diagnostics side, vastly more complicated inference, infinite amounts of fabricated narratives, and broader scope of controls on the authority side.
The PDF is linked from https://www.researchgate.net/publication/387051732_Estimatin...
> This study uses influenza mortality reduction (IMR) as an indicator of the aggregate effect of non-pharmaceutical interventions (NPI's) on the spread of respiratory infections to assess their impact on COVID mortality. [...] A counterfactual model of no NPI's suggests they prevented 850,000 COVID related deaths in the United States.
Off topic:
Strong disagree on the rest of what you said. It's also easy to forget the dangers of Covid pre-vaccine when we're quite deeply post-vaccine now.
This was without 'ai' but the blame was nearly close, "oops automated process did it!"
The 'artificial intelligence' has been here (starts with B). Kafka wrote about it comically.
A former roommate mentioned to me “regulations are often written in blood” and with the current atmosphere, I wonder if and when that might happen. The US political system has its flaws. Firearms are a serious public health hazard and constructive mediation is consistently passed over. It’s a far more primitive technology comparatively speaking.
Very glad this type of discussion is able to be framed without hysterics or other easily dismissed rhetoric.
The global monetary system has an uncanny ability to silently destroy people who sit on the periphery of politics. This is the perfect mechanism to destroy humanity... Keep silently taking out the people on the periphery, keep making the people in the center more greedy and narcissistic so they don't notice everyone falling all around them and keep supporting the current system... Eventually, those who were near the center will fall victim too, but by that point, they will have no power and no voice and the people left in the center will be yet more greedy and narcissistic and will basically not care.
I agree with your point, but per your own words that isn’t a big disaster but multiple little ones. Each person might have learned about some of them but only in passing and without too much information.
“The big one” will be something which everyone will be made aware of from the news, like the CrowdStrike outage.
"If you think about it - we're one misplaced negative sign, hallucinated by an LLM away from a disaster. At least multiply by zero won't crash things /s"
Lots has been written about this topic, https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-annou... is one of the more pessimistic takes. I wrote a more neutral introduction a while back at https://ruudvanasseldonk.com/2024/ai-alignment-starter-pack.