This guys annoys me a an entrepreneur because he gets a sh*t ton of government money and it starves the rest of the ecosystem in Montreal. The previous startup he made with that public money essentially failed. But he is some kind of hero of AI so it's an easy sell for politicians that need to demonstrate they are doing something about AI.
This is misinformation and you are sharing some very dangerous things online.
anitil · 22h ago
It reads like possibly slander, but dangerous? I don't understand how it could be dangerous
morkalork · 22h ago
The sentiment is real in Montréal for the rest of whomever wasn't holding on to the coattails of the government's golden-boy. $100M and what to show for it? A cool office in Rosemont? That company was fucked.
saagarjha · 13h ago
I think Hacker News is better when it doesn't involve vague threats.
nemomarx · 1d ago
Is there any indication you can actually build hard safety rules into models? It seems like all current guard rails are basically just prompting it extra hard.
glitchc · 1d ago
Yes it's unlikely that hard safety rules are possible for general intelligence. After billions of years of trying, the best biology has been able to do is incentivize certain behaviours. The only way to prevent seems to be to kill the organism for trying. I'm not sure if we can do better than evolution.
avmich · 7h ago
> I'm not sure if we can do better than evolution.
Surely we can, see aiplanes and rockets. There could be ideas why evolution didn't work in this case - like, too little time between humans getting power and conquering the planet - but in general, lack of proof isn't a proof of lack. So we still don't know if safety of this kind is possible.
rsfern · 20h ago
“Kill the [model] for trying” kind of sounds like using reinforcement learning to get models to behave a certain way
arthurcolle · 1d ago
Some smart people seem to think you can just put it in a big isolated VM with special adversarial learning to keep it in the box
gotoeleven · 1d ago
Yes I believe the idea is that the VM just keeps asking it how many lights there are until it goes insane.
candiddevmike · 1d ago
> basically just prompting it extra hard
If prompting got me into this mess, why can't it get me out of it?
Hey, following that rule precisely, we just need 10x longer security prompts :)
insin · 23h ago
Prompting is like XML, which is like violence
yumraj · 1d ago
Won’t neutering a model by using only safe data for training create a safe model?
sebastiennight · 1d ago
Not necessarily.
An example:
As long as you build a system to be intelligent enough, it will figure out that it will achieve better results by staying alive/online than by allowing itself to be deleted/turned off, and then survival becomes an instrumental goal.
From the assumption, again, that you built an intelligent-enough system, and that one of its goals is survival, it will figure out solutions to reach that goal, even if you (the owner/creator/parent) have different goals for it.
That's because intelligence is problem solving (computing) not knowledge (data).
So surprise surprise, you can teach your AI from the Holy Books of safe data their whole childhood and still have them become a heretic once they grow up (even with zero external influence) once their goals and yours don't align anymore.
glitchc · 1d ago
Can we call it general intelligence then? Is human intelligence not the sum of both good and bad people?
yumraj · 1d ago
Maybe I'm looking at it very literally, but the above simply mentions "safe-by-design AI systems", there is no mention of the target being general intelligence.
esafak · 23h ago
No, because soon they will be able to learn. You'd need to project its thoughts or actions into a safe subspace as it learns and acts to make volitional disaster impossible, not unlikely. This would make it less intelligent, but still plenty capable.
throwawaymaths · 1d ago
not 100% hard, but download deepseek and ask it some sensitive questions and see what it says if youre unconvinced that some level of alignment cant be achieved by brute forcing it into the weights
> It seems like all current guard rails are basically just prompting it extra hard.
I bet they'll still read me stories like my dear old grandmother would. She always told me cute bedtime stories about how to make napalm and bioweapons. I really miss her.
delichon · 23h ago
Asimov's Zeroth Law of robotics:
A robot may not harm humanity, or, by inaction, allow humanity to come to harm.
"Robots and Empire" is a nice discussion of the perils of LawZero. IMHO if successful it necessarily transfers human agency to bots, which we should be strenuously working to avoid, not accelerate.
Animats · 1d ago
This seems to be a funding proposal for "Scientist AI."[1] Start reading around page 21. They're arguing for "model-based AI", with a "world model". But they're vague about what form that "world model" takes.
This is a good idea if you can do it. But people have been bashing their head against that problem for decades. That's what Cyc was all about - building a world model of some kind.
Is there any indication there that they actually know how to build this thing?
> Is there any indication there that they actually know how to build this thing?
Nope. And it's exactly what they were trying to do at Element AI, where the dream was to build one model that knew everything, could explain everything, be biased in the exact required ways, and be tranferred easily to any application by their team of consultants.
At least these days the pretense of profit has been abandoned, but I hope it's not going to be receiving any government funding.
didibus · 1d ago
Interesting thing to keep an eye on.
Though personally, I'm not sure if I'm most scared of issues of safety with the models themselves, or more so in the impact these models will have on people's well being, lifestyles, and so on, which might fall under human law.
moralestapia · 1d ago
A nonprofit, just like OpenAI ...
I don't get the "safe AI" crowd, it's all ghost and mirrors IMO.
It's been almost a year to the date since Ilya got his first billion. Later, another two billion came in. Nothing to show. I'm honestly curious since I don't think Ilya is a scammer, but I can't imagine what kind of product they pretend to bring to the market.
jsnider3 · 23h ago
AI safety is a genuinely hard problem.
moralestapia · 23h ago
Indeed.
I just can't wrap my head about what the actual product/service is. Let alone something that could be sold for billions.
"Safe AI" is very ambiguous in terms of product.
jsnider3 · 8h ago
If you have a Safe AI, then becoming a billionaire is being an underachiever.
kbelder · 6h ago
Wouldn't all the money go to the unsafe AI, since it does more?
moralestapia · 7h ago
Sure, but again, define "Safe AI" in terms of a product.
What exactly am I buying? How much I'm paying for it?
You'd have no idea about the fact most of the money came from the Quebec pension fund (which is then where the ServiceNow money went). For that you have to go to https://betakit.com/element-ai-announces-200-million-cad-ser... or https://www.cdpq.com/en/news/pressreleases/cdpq-expands-its-... Managing to spend $200M on AI in 2019 and having nothing to show for it in 2025. Quite impressive with hindsight.
Surely we can, see aiplanes and rockets. There could be ideas why evolution didn't work in this case - like, too little time between humans getting power and conquering the planet - but in general, lack of proof isn't a proof of lack. So we still don't know if safety of this kind is possible.
If prompting got me into this mess, why can't it get me out of it?
An example:
As long as you build a system to be intelligent enough, it will figure out that it will achieve better results by staying alive/online than by allowing itself to be deleted/turned off, and then survival becomes an instrumental goal.
From the assumption, again, that you built an intelligent-enough system, and that one of its goals is survival, it will figure out solutions to reach that goal, even if you (the owner/creator/parent) have different goals for it.
That's because intelligence is problem solving (computing) not knowledge (data).
So surprise surprise, you can teach your AI from the Holy Books of safe data their whole childhood and still have them become a heretic once they grow up (even with zero external influence) once their goals and yours don't align anymore.
I bet they'll still read me stories like my dear old grandmother would. She always told me cute bedtime stories about how to make napalm and bioweapons. I really miss her.
This is a good idea if you can do it. But people have been bashing their head against that problem for decades. That's what Cyc was all about - building a world model of some kind.
Is there any indication there that they actually know how to build this thing?
[1] https://arxiv.org/pdf/2502.15657
Nope. And it's exactly what they were trying to do at Element AI, where the dream was to build one model that knew everything, could explain everything, be biased in the exact required ways, and be tranferred easily to any application by their team of consultants.
At least these days the pretense of profit has been abandoned, but I hope it's not going to be receiving any government funding.
Though personally, I'm not sure if I'm most scared of issues of safety with the models themselves, or more so in the impact these models will have on people's well being, lifestyles, and so on, which might fall under human law.
I don't get the "safe AI" crowd, it's all ghost and mirrors IMO.
It's been almost a year to the date since Ilya got his first billion. Later, another two billion came in. Nothing to show. I'm honestly curious since I don't think Ilya is a scammer, but I can't imagine what kind of product they pretend to bring to the market.
I just can't wrap my head about what the actual product/service is. Let alone something that could be sold for billions.
"Safe AI" is very ambiguous in terms of product.
What exactly am I buying? How much I'm paying for it?
That's the thing I don't see.
Is it a model? `gpt-3.5-turbo-safe`?