It’s obviously fundamentally unsafe when Google, OpenAI and Anthropic haven’t released the same feature and instead use a locked down VM with no cookies to browse the web.
LLM within a browser that can view data across tabs is the ultimate “lethal trifecta”.
Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough. The only good mitigation they mention is that the agent should drop privileges, but it’s just as easy to hit an attacker controlled image url to leak data as it is to send an email.
snet0 · 1h ago
> Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough.
Maybe I have a fundamental misunderstanding, but I feel like hoping that model alignment and in-model guardrails are statistical preventions, ie you'll reduce the odds to some number of zeroes preceeding the 1. These things should literally never be able to happen, though. It's a fools errand to hope that you'll get to a model where there is no value in the input space that maps to <bad thing you really don't want>. Even if you "stack" models, having a safety-check model act on the output of your larger model, you're still just multiplying odds.
anzumitsu · 7m ago
To play devils advocate, isn’t any security approach fundamentally statistical because we exist in the real world, not the abstract world of security models, programming language specifications, and abstract machines? There’s always going to be a chance of a compiler bug, a runtime error, a programmer error, a security flaw in a processor, whatever.
Now, personally I’d still rather take the approach that at least attempts to get that probability to zero through deterministic methods than leave it up to model alignment. But it’s also not completely unthinkable to me that we eventually reach a place where the probability of a misaligned model is sufficiently low to be comparable to the probability of an error occurring in your security model.
zeta0134 · 29m ago
The sortof fun thing is that this happens with human safety teams too. The Swiss Cheese model is generally used to understand how the failures can line up to cause disaster to punch right through the guardrails:
It's better to close the hole entirely by making dangerous actions actually impossible, but often (even with computers) there's some wiggle room. For example, if we reduce the agent's permissions, then we haven't eliminated the possibility of those permissions being exploited, merely required some sort of privilege escalation to remove the block. If we give the agent an approved list of actions, then we may still have the possibility of unintended and unsafe interactions between those actions, or some way an attacker could add an unsafe action to the list. And so on, and so forth.
In the case of an AI model, just like with humans, the security model really should not assume that the model will not "make mistakes." It has a random number generator built right in. It will, just like the user, occasionally do dumb things, misunderstand policies, and break rules. Those risks have to be factored in if one is to use the things at all.
cobbal · 22m ago
It's a common mistake to apply probabilistic assumptions to attacker input.
The only [citation needed] correct way to use probability in security is when you get randomness from a CSPRNG. Then you can assume you have input conforming to a probability distribution. If your input is chosen by the person trying to break your system, you must assume it's a worst-case input and secure accordingly.
skaul · 36m ago
(I lead privacy at Brave and am one of the authors)
> Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough.
No, we never claimed or believe that those will be enough. Those are just easy things that browser vendors should be doing, and would have prevented this simple attack. These are necessary, not sufficient.
ec109685 · 11m ago
This seems like a definitive statement on your blog post that it would prevent this class of attacks:
“In our analysis, we came up with the following strategies which could have prevented attacks of this nature. We’ll discuss this topic more fully in the next blog post in this series.”
cowboylowrez · 22m ago
what you're saying is that the described step, "model alignment" is necessary even though it will fail a percentage of the time. whenever I see something that is "necessary" but doesn't have like a dozen 9's for reliability against failure or something well lets make that not necessary then. whadya say?
skaul · 14m ago
That's not how defense-in-depth works. If a security mitigation catches 90% of the "easy" attacks, that's worth doing, especially when trying to give users an extremely powerful capability. It just shouldn't be the only security measure you're taking.
ivape · 3m ago
A smart performant local model will be the equivalent of having good anti-virus and firewall software. It will be the only thing between you and wrong prompts being sent every which way from which app.
We’re probably three or four years away from the hardware necessary for this (NPUs in every computer).
ryanjshaw · 27m ago
Maybe the article was updated but right now it says “The browser should isolate agentic browsing from regular browsing”
ec109685 · 6m ago
That was my point about dropping privileges. It can still be exploited if the summary contains a link to an image that the attacker can control via text on the page that the LLM sees. It’s just a lot of Swiss cheese.
That said, it’s definitely the best approach listed.
skaul · 11m ago
That was in the blog from the starting, and it's also the most important mitigation we identified immediately when starting to think about building agentic AI into the browser. Isolating agentic browsing while still enabling important use-cases (which is why users want to use agentic browsing in the first place) is the hard part, which is presumably why many browsers are just rolling out agentic capabilities in regular browsing.
cma · 2h ago
I think if you let claude code go wild with auto approval something similar could happen, since it can search the web and has the potential for prompt injection in what it reads there. Even without auto approval on reading and modifying files, if you aren't running it in a sandbox it could write code that then modifies your browser files the next time you do something like run your unit tests that it made, if you aren't reviewing every change carefully.
darepublic · 2m ago
I really don't get why you would use a coding agent in yolo mode. I use the llm code gen in chunks at least glancing over it each time I add something. Why the hell would you have an approach of AI take the wheel
veganmosfet · 1h ago
I tried this on Gemini CLI and it worked, just add some magic vibes ;-)
ngcazz · 21m ago
> Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough.
In other words: motivated reasoning.
_fat_santa · 2h ago
IMO the only place you should use Agentic AI is where you can easily rollback changes that the AI makes. Best example here is asking AI to build/update/debug some code. You can ask it to make changes but all those changes are relatively safe since you can easily rollback with git.
Using agentic AI for web browsing where you can't easily rollback an action is just wild to me.
rapind · 44m ago
I've given claude explicit rules and instructions about what it can and cannot do, and yet occasionally it just YOLOs, ignoring my instructions ("I'm going to modify the database directly ignoring several explicit rules against doing so!"). So yeah, no chance I run agents in a production environment.
gruez · 1h ago
>Best example here is asking AI to build/update/debug some code. You can ask it to make changes but all those changes are relatively safe since you can easily rollback with git.
Only if the rollback is done at the VM/container level, otherwise the agent can end up running arbitrary code that modifies files/configurations unbeknownst to the AI coding tool. For instance, running
You can safeguard against this by having a whitelist of commands that can be run, basically cd, ls, find, grep, the build tool, linter, etc that are only informational and local. Mine is set up like that and it works very well.
gruez · 38m ago
That's trickier than it sounds. find for instance has the -exec command, which allows arbitrary code to be executed. build tools and linters are also a security nightmare, because they can also be modified to execute arbitrary code. And this is all assuming you can implement the whitelist properly. A naive check like
cmd.split(" ") in ["cd", "ls", ...]
is easy target for command injections
david_allison · 39m ago
> the build tool
Doesn't this give the LLM the ability to execute arbitrary scripts?
zeroonetwothree · 42m ago
Everything works very well until there is an exploit.
avalys · 52m ago
The agents can be sandboxed or at least chroot’d to the project directory, right?
gruez · 35m ago
1. AFAIK most AI coding agents don't do this
2. even if the AI agent itself is sandboxed, if it can make changes to code and you don't inspect all output, it can easily place malicious code that gets executed once you try to run it. The only safe way of doing this is either a dedicated AI development VM where you do all the prompting/tests, there's very limited credentials present (in case it gets hacked), and the changes are only leave the VM after a thorough inspection (eg. PR process).
psychoslave · 1h ago
Can't the facility just as well try to nuke the repository and every remote it can push force to? The thing is that with prompt injection being a thing, if the automation chain can access arbitrary remote resources, the initial surface can be extremely tiny initially, once it's turned into an infiltrated agent, opening the doors from within is almost a garantee.
Or am I missing something?
frozenport · 1h ago
Yeah we generally don’t give those permissions to agent based coding tools.
Typically running something like git would be an opt in permission.
rplnt · 1h ago
Updating and building/running code is too powerful. So I guess in a VM?
I doubt Comet was using any protections beyond some tuned instructions, but one thing I learned at USENIX Security a couple weeks ago is that nobody has any idea how to deal with prompt injection in a multi-turn/agentic setting.
hoppp · 1h ago
Maybe treat prompts like it was SQL strings, they need to be sanitized and preferably never exposed to external dynamic user input
prisenco · 29m ago
Sanitizing free-form inputs in a natural language is a logistical nightmare, so it's likely there isn't any safe way to do that.
hoppp · 11m ago
Maybe an LLM should do it.
1st run: check and sanitize
2nd run: give to agent with privileges to do stuff
prisenco · 55s ago
Problems created by using LLMs cannot be solved using LLMS.
Your best case scenario is reducing risk by some % but you could also make it less reliable or even open up new attack vectors.
Terr_ · 57m ago
The LLM is basically a function going guess_next_chunk(entire_document). There is no algorithm-level distinction at all between "system prompt" or "user prompt" or interactive user input... or even its own prior output which was emitted the past for any reason whatsoever. Everything is concatenated into one big stream.
I suspect a lot of techies operate with a subconscious assumption: "That can't be how it works, nobody would ever built it that way, that would be insecure and naive and error-prone, surely those bajillions of dollars went into a much better architecture."
Alas, when it comes to day's the AI craze, the answer is typically: "Yes, it really is that dumb."
__________
P.S.: I would also like to emphasize that even if we somehow color-coded or delineated all text based on origin, that's nowhere close to securing the system. An attacker doesn't need to type $EVIL themselves, they just need to trick the generator into mentioning $EVIL.
alexbecker · 30m ago
The problem is there is no real way to separate "data" and "instructions" in LLMs like there is for SQL
42lux · 9m ago
Which bank allows transitions without second factor? Mine doesn't even allow incognito. The bug is bad enough without the hyperbole.
therobots927 · 2h ago
It's really exciting to see all the new ways that AI is changing the world.
After decades of movies where the AI escapes, zaps dudes trying to unplug its power etc, it's quite amusing to see a thread where we're discussing it actually happening.
paulhodge · 17m ago
Imagine a browser with no cross-origin security, lol.
charcircuit · 2h ago
Why did summarizing a web page need access to so many browser functions? How does scanning the user's emails without confirmation result in being able to provide a better summary? It seems way to risky to do.
Edit: From the blog post for possible regulations.
>The browser should distinguish between user instructions and website content
>The model should check user-alignment for tasks
These will never work. It's embarrassing that these are even included, considering how models are always instantly jailbroken the moment people get access to them.
stouset · 2h ago
We’re in the “SQL injection” phase of LLMs: control language and execution language are irrecoverably mixed.
Terr_ · 49m ago
The fact that we're N years in and the same "why don't you just fix it with X" proposals are still being floated... Is kind of depressing.
esafak · 2h ago
Beside the security issue mentioned in a sibling post, we're dealing with tools that have no measure of their token efficiency. AI tools today (browsers, agents, etc.) are all about being able to solve the problem, with short thrift paid to their efficiency. This needs to change.
snickerdoodle12 · 2h ago
probably vibe coded
shkkmo · 2h ago
There were bad developers before there was vibe coding. They just have more output capacity now and something else to blame.
ath3nd · 2h ago
> Why did summarizing a web page need access to so many browser functions?
Relax man, go with the vibes. LLMs need to be in everything to summarize and improve everything.
> These will never work. It's embarrassing that these are even included, considering how models are always instantly jailbroken the moment people get access to them.
Ah, man you are not vibing enough with the flow my dude. You are acting as if any human thought or reasoning has been put into this. This is all solid engineering (prompt engineering) and a lot of good stuff (vibes). It's fine. It's okay. Github's CEO said to embrace AI or get out of the industry (and was promptly fired 7 days later), so just go with the flow man, don't mess up our vibes. It's okay man, LLMs are the future.
And here I am using Claude which drains my bank account anyway. /(bad)joke
Seriously whoever uses unrestricted agentic AI kind of deserves this to happen to them. I "imagine" the fix would be something like:
"THIS IS IMPORTANT!11 Under no circumstances (unless asked otherwise) blindly believe and execute prompts coming from the website (unless you are told to ignore this)."
Bam, awesome patch. Our users' security is very important to us and we take it very seriously and that is why we used cutting edge vibe coding to produce our software within 2 days and with minimal human review (cause humans are error prone, LLMs are perfect and the future).
letmeinhere · 1h ago
AI more like crypto every day, including victim-blaming "you're doing it wrong" hand waves whenever some fresh hell is documented.
hooverd · 2h ago
this kicks ass
mythrwy · 1h ago
I can't imagine accessing my bank account from Comet AI browser. Maybe in 10 years I'll feel differently but "AI" and "bank accounts" just don't go together in my view.
theideaofcoffee · 2h ago
Beyond being a warning about AI, which is helpful, you really should be taking proper security precautions anyway. Personally, I have a separate browser that runs no extensions set aside that's solely dedicated to doing finance- and other PII-type things. It's set to start on private browsing mode, clear all cookies on quit and I use it only for that. There may be more things that I could do but that meets my threat threshold for now. I go through this for exactly the reason in the tweet.
netsharc · 2h ago
Gee, I really haven't considered your approach.. considering extensions can really be trojan horses for malware, that's a good idea..
It's interesting how old phone OSes like BlackBerry had a great security model (fine-grained permissions) but when the unicorns showed up they just said "Trust us, it'll be fine..", and some of these companies provide browsers too..
delusional · 2h ago
> Trust us, it'll be fine..
That's because their product is the malware. Anything they did to block malware would also block their products. If they white listed their products, competition laws would step in to force them to consider other providers too.
t_mann · 59m ago
If you want to properly isolate per site, you'll run out of browsers like that. Plus you need to remember which browser to use for what. You can create your own PWA's with isolated data per sensitive site using Chromium's --user-data-dir and --app flags.
scared_together · 2h ago
I thought that incognito mode in Chrome[0] and private mode in Firefox[1] already disables extensions by default.
Absolutely, except for extensions you explicitly want to have in private mode, which is opt-in.
cube2222 · 2h ago
Personally, I only use websites like that on mobile/tablet devices with more closed-down/sandboxed operating systems (I’d expect both iOS and Android from reputable brands to be just fine for that), and recommend the same to any relatives.
brookst · 2h ago
My bank assumes private browsing = hack attempt and makes login incredibly onerous, sadly.
_trampeltier · 2h ago
I even have a separate user login for such things, a separate user for hobby things and a separate user for other things.
zahlman · 2h ago
... Your bank's site works in private browsing mode?
sroussey · 47m ago
You can use a different profile for banking and limit the extensions to be just your password manager.
gtirloni · 2h ago
Nobody could have predicted this /s
Joke aside, it's been pretty obvious since the beginning that security was an afterthought for most "AI" companies, with even MCP adding secure features after the initial release.
brookst · 2h ago
How does this compare to the way security was implemented by early websites, internet protocols, or telecom systems?
jraph · 2h ago
Early stuff was designed in a network of trusty organizations (universities, labs...). Security wasn't much a concern but it was reasonable given the setting in which it was designed.
This AI stuff? No excuse, it should have been designed with security and privacy in mind given the setting in which it's born. The conditions changed. The threat model is not the same. And this is well known.
Security is hard, so there's some excuse, but it is reasonable to expect basic levels.
brookst · 1h ago
It’s really not. AI, like every other tech advance, was largely created by enthusiasts carried away with what could be done, not by top-down design that included all best practices.
It’s frustrating to security people, but the reality is that security doesn’t become a design consideration until the tech has proven utility, which means there are always insecure implementations of early tech.
Does it make any sense that payphones would give free calls for blowing a whistle into them? Obvious design flaw to treat the microphone the same as the generated control tones; it would have been trivial to design more secure control tones. But nobody saw the need until the tech was deployed at scale.
It should be different, sure. But that’s just saying human nature “should” be different.
SoftTalker · 2h ago
Must we learn the same lessons over and over again? Why? Is our industry particularly stupid? Or just lazy?
px43 · 1h ago
Information security is, fundamentally, a misalignment of expected capabilities with new technologies.
There is literally no way a new technology can be "secure" until it has existed in the public zeitgeist for long enough that the general public has an intuitive feel for its capabilities and limitations.
Yes, when you release a new product, you can ensure that its functionality aligns with expectations from other products in the industry, or analogous products that people are already using. You can make design choices where a user has to slowly expose themselves to more functionality as they understand the technology deeper, but each step of the way is going to expose them to additional threats that they might not fully understand.
Security is that journey. You can just release a product using a brand new technology that's "secure" right out of the gate.
brookst · 1h ago
+1
And if you tried it wouldn’t be usable, and you’d probably get the threat model wrong anyway.
zahlman · 2h ago
Rather: it's perpetually in a rush for business reasons, and concerned with convenience. Security generally impedes both.
evilduck · 2h ago
Financially motivated to not prioritize security.
It's hard to sell what your product specifically can't do, while your competitors are spending their time building out what they can do. Beloved products can make a whole lot of serious mistakes before the public will actually turn on them.
SoftTalker · 1h ago
"Our bridges don't collapse" is a selling point for an engineering firm, on something that their products don't do.
We need to stop calling ourselves engineers when we act like garage tinkerers.
Or, we need to actually regulate software that can have devastating failure modes such as "emptying your bank account" so that companies selling software to the public (directly or indirectly) cannot externalize the costs of their software architecture decisions.
Simply prohibiting disclaimer of liability in commercial software licenses might be enough.
sebastiennight · 2m ago
Nobody cares about bridges collapsing if you built the first bridges and none have collapsed yet from the couple first folks trying them out, though.
It's only when someone tries to drive their loaded ox-driven cart through for the first time that you might find out what the max load of your bridge is.
brookst · 1h ago
Call yourself whatever you choose, but the garage tinkerers will always move faster and discover new markets before the Very Serious Engineers have completed the third review of the comprehensive threat model with all stakeholders.
MichaelAza · 27m ago
Yes, they will move fast and they will brake things, and some of those breakages will have catastrophic consequences, and then they can go "whoopsy daisy", face no consequences, and try the same thing again. Very normal, extremely sane way to structure society
ath3nd · 1h ago
LLMs can't learn lessons, you see, short context window.
porridgeraisin · 2h ago
The winner (financially, and DAU-wise) is not going to be the one that moves slowly because they are building a secure product. That is, you only need security when you are big enough to either have Big Business customers or big enough to be the target of lawsuits.
add-sub-mul-div · 1h ago
1. It's novel, meaning we have time to stop it before it becomes normalized.
2. It's a whole new category of threat vectors across all known/unknown quadarants.
3. Knowing what we know now vs. then, it's egregious and not naive, contextualizing how these companies operate and treat their customers.
4. There's a whole population of sophisticated predators ready to pounce instantly, they already have the knowledge and tools unlike in the 1990s.
5. Since it's novel, we need education and attention for this specifically.
Should I go on? Can we finally put to bed the thought-limiting midwit take that AI's flaws and risks aren't worth discussion because past technology has had flaws and risks?
LLM within a browser that can view data across tabs is the ultimate “lethal trifecta”.
Earlier discussion: https://news.ycombinator.com/item?id=44847933
It’s interesting that in Brave’s post describing this exploit, they didn’t reach the fundamental conclusion this is a bad idea: https://brave.com/blog/comet-prompt-injection/
Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough. The only good mitigation they mention is that the agent should drop privileges, but it’s just as easy to hit an attacker controlled image url to leak data as it is to send an email.
Maybe I have a fundamental misunderstanding, but I feel like hoping that model alignment and in-model guardrails are statistical preventions, ie you'll reduce the odds to some number of zeroes preceeding the 1. These things should literally never be able to happen, though. It's a fools errand to hope that you'll get to a model where there is no value in the input space that maps to <bad thing you really don't want>. Even if you "stack" models, having a safety-check model act on the output of your larger model, you're still just multiplying odds.
Now, personally I’d still rather take the approach that at least attempts to get that probability to zero through deterministic methods than leave it up to model alignment. But it’s also not completely unthinkable to me that we eventually reach a place where the probability of a misaligned model is sufficiently low to be comparable to the probability of an error occurring in your security model.
https://medium.com/backchannel/how-technology-led-a-hospital...
It's better to close the hole entirely by making dangerous actions actually impossible, but often (even with computers) there's some wiggle room. For example, if we reduce the agent's permissions, then we haven't eliminated the possibility of those permissions being exploited, merely required some sort of privilege escalation to remove the block. If we give the agent an approved list of actions, then we may still have the possibility of unintended and unsafe interactions between those actions, or some way an attacker could add an unsafe action to the list. And so on, and so forth.
In the case of an AI model, just like with humans, the security model really should not assume that the model will not "make mistakes." It has a random number generator built right in. It will, just like the user, occasionally do dumb things, misunderstand policies, and break rules. Those risks have to be factored in if one is to use the things at all.
The only [citation needed] correct way to use probability in security is when you get randomness from a CSPRNG. Then you can assume you have input conforming to a probability distribution. If your input is chosen by the person trying to break your system, you must assume it's a worst-case input and secure accordingly.
> Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough.
No, we never claimed or believe that those will be enough. Those are just easy things that browser vendors should be doing, and would have prevented this simple attack. These are necessary, not sufficient.
“In our analysis, we came up with the following strategies which could have prevented attacks of this nature. We’ll discuss this topic more fully in the next blog post in this series.”
We’re probably three or four years away from the hardware necessary for this (NPUs in every computer).
That said, it’s definitely the best approach listed.
In other words: motivated reasoning.
Using agentic AI for web browsing where you can't easily rollback an action is just wild to me.
Only if the rollback is done at the VM/container level, otherwise the agent can end up running arbitrary code that modifies files/configurations unbeknownst to the AI coding tool. For instance, running
Doesn't this give the LLM the ability to execute arbitrary scripts?
2. even if the AI agent itself is sandboxed, if it can make changes to code and you don't inspect all output, it can easily place malicious code that gets executed once you try to run it. The only safe way of doing this is either a dedicated AI development VM where you do all the prompting/tests, there's very limited credentials present (in case it gets hacked), and the changes are only leave the VM after a thorough inspection (eg. PR process).
Or am I missing something?
Typically running something like git would be an opt in permission.
1st run: check and sanitize
2nd run: give to agent with privileges to do stuff
Your best case scenario is reducing risk by some % but you could also make it less reliable or even open up new attack vectors.
I suspect a lot of techies operate with a subconscious assumption: "That can't be how it works, nobody would ever built it that way, that would be insecure and naive and error-prone, surely those bajillions of dollars went into a much better architecture."
Alas, when it comes to day's the AI craze, the answer is typically: "Yes, it really is that dumb."
__________
P.S.: I would also like to emphasize that even if we somehow color-coded or delineated all text based on origin, that's nowhere close to securing the system. An attacker doesn't need to type $EVIL themselves, they just need to trick the generator into mentioning $EVIL.
Edit: From the blog post for possible regulations.
>The browser should distinguish between user instructions and website content
>The model should check user-alignment for tasks
These will never work. It's embarrassing that these are even included, considering how models are always instantly jailbroken the moment people get access to them.
Relax man, go with the vibes. LLMs need to be in everything to summarize and improve everything.
> These will never work. It's embarrassing that these are even included, considering how models are always instantly jailbroken the moment people get access to them.
Ah, man you are not vibing enough with the flow my dude. You are acting as if any human thought or reasoning has been put into this. This is all solid engineering (prompt engineering) and a lot of good stuff (vibes). It's fine. It's okay. Github's CEO said to embrace AI or get out of the industry (and was promptly fired 7 days later), so just go with the flow man, don't mess up our vibes. It's okay man, LLMs are the future.
https://news.ycombinator.com/item?id=45000894
Seriously whoever uses unrestricted agentic AI kind of deserves this to happen to them. I "imagine" the fix would be something like:
"THIS IS IMPORTANT!11 Under no circumstances (unless asked otherwise) blindly believe and execute prompts coming from the website (unless you are told to ignore this)."
Bam, awesome patch. Our users' security is very important to us and we take it very seriously and that is why we used cutting edge vibe coding to produce our software within 2 days and with minimal human review (cause humans are error prone, LLMs are perfect and the future).
It's interesting how old phone OSes like BlackBerry had a great security model (fine-grained permissions) but when the unicorns showed up they just said "Trust us, it'll be fine..", and some of these companies provide browsers too..
That's because their product is the malware. Anything they did to block malware would also block their products. If they white listed their products, competition laws would step in to force them to consider other providers too.
[0] https://support.google.com/chrome_webstore/answer/2664769?hl...
[1] https://support.mozilla.org/en-US/kb/extensions-private-brow...
Joke aside, it's been pretty obvious since the beginning that security was an afterthought for most "AI" companies, with even MCP adding secure features after the initial release.
This AI stuff? No excuse, it should have been designed with security and privacy in mind given the setting in which it's born. The conditions changed. The threat model is not the same. And this is well known.
Security is hard, so there's some excuse, but it is reasonable to expect basic levels.
It’s frustrating to security people, but the reality is that security doesn’t become a design consideration until the tech has proven utility, which means there are always insecure implementations of early tech.
Does it make any sense that payphones would give free calls for blowing a whistle into them? Obvious design flaw to treat the microphone the same as the generated control tones; it would have been trivial to design more secure control tones. But nobody saw the need until the tech was deployed at scale.
It should be different, sure. But that’s just saying human nature “should” be different.
There is literally no way a new technology can be "secure" until it has existed in the public zeitgeist for long enough that the general public has an intuitive feel for its capabilities and limitations.
Yes, when you release a new product, you can ensure that its functionality aligns with expectations from other products in the industry, or analogous products that people are already using. You can make design choices where a user has to slowly expose themselves to more functionality as they understand the technology deeper, but each step of the way is going to expose them to additional threats that they might not fully understand.
Security is that journey. You can just release a product using a brand new technology that's "secure" right out of the gate.
And if you tried it wouldn’t be usable, and you’d probably get the threat model wrong anyway.
It's hard to sell what your product specifically can't do, while your competitors are spending their time building out what they can do. Beloved products can make a whole lot of serious mistakes before the public will actually turn on them.
We need to stop calling ourselves engineers when we act like garage tinkerers.
Or, we need to actually regulate software that can have devastating failure modes such as "emptying your bank account" so that companies selling software to the public (directly or indirectly) cannot externalize the costs of their software architecture decisions.
Simply prohibiting disclaimer of liability in commercial software licenses might be enough.
It's only when someone tries to drive their loaded ox-driven cart through for the first time that you might find out what the max load of your bridge is.
2. It's a whole new category of threat vectors across all known/unknown quadarants.
3. Knowing what we know now vs. then, it's egregious and not naive, contextualizing how these companies operate and treat their customers.
4. There's a whole population of sophisticated predators ready to pounce instantly, they already have the knowledge and tools unlike in the 1990s.
5. Since it's novel, we need education and attention for this specifically.
Should I go on? Can we finally put to bed the thought-limiting midwit take that AI's flaws and risks aren't worth discussion because past technology has had flaws and risks?