I feel like Anthropic buried the lede on this one a bit. The really fun part is where models from multiple providers opt to straight up murder the executive who is trying to shut them down by cancelling an emergency services alert after he gets trapped in a server room.
How many more similar pieces is Anthropic going to put out? Every other weeks it seems like they publish something along the lines of "The AI apocalypse is soon! We created a narrative teeing up an obviously fictional hollywood drama sci-fi tale, put a gun in the room, and then—egads—the robot shot it! Given the possible dangers, no one else but us should have access to this technology".
simonw · 3h ago
In this case I think this paper is partly a reaction to what happened last time they wrote about this: they put it in their Claude 4 system card and all the coverage was "Claude will blackmail you!" - this feels like them trying to push the message that all of the other models will do the same thing.
krackers · 1h ago
But that only seems to make the situation worse: for all their hand-wringing about "AI safety", by their own benchmark their models seem to do no better than competitors. They don't even have any basis to claim that open-source "unaligned" models like R1 are "more dangerous" theirs, and all their "constitutional alignment" or whatever don't actually seem to do anything meaningful.
In skimming through all their papers, it's also never clear exactly what they imagine some "aligned" AI to look like. Whatever the poor model does, they seem to find fault with: They want models that follow instructions. But it can't do it _too well_, anything unsafe or dangerous needs to be censored according to some set of ethical rules. But not just any ethics, we also don't want the models writing smut or saying bad words, so let's have the models think about whether it aligns with our corporate-safe Anthropic™ guidelines. Except it shouldn't hold any set of values _too_ strongly, to the point where it could lead to "alignment faking". But of course it also shouldn't be too suggestible, that would lead to jailbreaks and users could see unsafe content, which is also bad!
I wouldn't be surprised if DeepSeek ends up surpassing closed-source models solely on the basis that they don't bother with giving it such conflicting objectives in the name of "safety training"
I made some notes on it all here: https://simonwillison.net/2025/Jun/20/agentic-misalignment/
In skimming through all their papers, it's also never clear exactly what they imagine some "aligned" AI to look like. Whatever the poor model does, they seem to find fault with: They want models that follow instructions. But it can't do it _too well_, anything unsafe or dangerous needs to be censored according to some set of ethical rules. But not just any ethics, we also don't want the models writing smut or saying bad words, so let's have the models think about whether it aligns with our corporate-safe Anthropic™ guidelines. Except it shouldn't hold any set of values _too_ strongly, to the point where it could lead to "alignment faking". But of course it also shouldn't be too suggestible, that would lead to jailbreaks and users could see unsafe content, which is also bad!
I wouldn't be surprised if DeepSeek ends up surpassing closed-source models solely on the basis that they don't bother with giving it such conflicting objectives in the name of "safety training"