I remember reading somewhere that Yudkowski said that he had been convinced to "let the AI out of the box" in a conversation with someone, or maybe it was the other way around, but either way the convincing arguments were not revealed.
This feels like the same kind of vague "rational mysticism." "We don't know what we don't know, and we're such silly humans, therefore...AI will kill us all" is all I can really take from it.
strken · 35m ago
Wasn't there an experiment where they had two groups of people, one which would be paid if they didn't let the AI out of the box, and nobody in that group actually let it out?
I really seriously doubt that an AI can convince a normal person to let it out, if they know they'll have their pay docked and if the communication is over text. The best scammers in the world can't convince most people to click on a link over text, let alone if "not clicking any links" was someone's job title.
baxtr · 2h ago
"rational mysticism" - what a great term to describe this genre
chasil · 3h ago
"There are two great tragedies in life: not getting what you want, and getting it."
- Oscar Wilde
globalnode · 2h ago
Because we're conditioned like the dog that chases the car. Its a good observation.
nis0s · 2h ago
Adversarial relationships might occur under any number of circumstances, but there’s a spectrum of this type of relationship where you have friendly competition at one end (healthy, pro-social) and detached vendetta at the other (unhealthy, antisocial).
When you come across an adversary, it’s to your benefit to try to bring them to the healthy side of things.
People can be pretty reasonable, and if they’re not then they can be shamed into behaving. If they cannot be shamed, then there’s retribution. If that doesn’t work, then there’s always the option to go full Rambo.
You never want to go full Rambo, but your adversary must understand that it’s an option that’s available to you. I don’t think super AI will be any different as an adversary, but maybe there’s something I haven’t considered.
Mistletoe · 2h ago
> Paul Crowley recently mentioned that we underrate the effect of the Russian IRA (Internet Research Agency) which works full-time on creating discord and anger among Americans online
What would a task force built to oppose the IRA seeding discord online look like? How would it operate? We need that.
01HNNWZ0MV43FF · 1h ago
My 2 cents is, people should be online less. This is vague scattershot advice but it would at least help me
andyferris · 2h ago
I suppose this is taken seriously by a society at war. Otherwise we tend to try to be civil (which requires giving the benefit of the doubt).
The white-anting by Russia hasn't really triggered this kind of "immune response" - it's hard to know what to do about it, which is of course the entire point.
politelemon · 3h ago
This reads like an exercise in deliberately misunderstanding the word adversary.
dang · 2h ago
Ok, but please don't post putdowns to Hacker News.
If you know more than others, that's great, but in that case please share some of what you know, so the rest of us can learn.
habitue · 2h ago
Maybe chess was wrong, it's adversarial. But there's definitely a qualitative leap between "adversary withing defined rules in a particular context" and "anything is on the table, they could kill you etc." kind of adversary
mgaunard · 2h ago
Any kind of competition is adversary; it's just that in most competitions, you'd be disadvantaged if it came to light that you didn't follow good sportmanship.
Smaug123 · 40m ago
I disagree! I claim that most people follow rules out of a general sense of fair play rather than because they will be punished for not doing so. Certainly this is true of me, and I don't believe society would look the way it does if the cynical view were nearly universally correct.
Cheaters in games like Magic are very rare; if most people tried to cheat whenever they thought they could get away with it, we'd be forced to set up competitions with more stringent verifiable rules like (off the top of my head) "all cards must be drawn and given to you from your deck by the opponent". We haven't done that, so I infer that most people don't try to cheat.
(In your favour, I do concede that everyone writes down their individual understanding of the history of a given chess game; but there are weak instrumental reasons for that even if people aren't cheating, because it is possible to upset a board by accident.)
XorNot · 2h ago
I'm not sure I really see that problem with it? It's a correct observation that people tend to discount what actual, intelligent opposition will do.
The number of people who declare they can totally trust what an adversary says because they agree with it is astounding, as though a committed opponent wouldn't do anything if it gained advantage including feinting in a way which seems unadvantegeous to gain long term advantage.
maxbond · 2h ago
There is however the flip side, where people distrust something because they believe an adversary said it. Sort of like how link spammers switched from SEO to "negative SEO" where, after Google started identifying and penalizing SEO networks, spammers started extorting people with the threat of linking to their site (thus penalizing them in search results). Blind trust and blind distrust are equally exploitable.
In the end, the only winning move is not to play. If you believe an adversary said something (or "a liar" if you prefer), you ignore it entirely. You make your mind up about what you believe based on evidence, and you decide if you agree with someone based on how well their statement comports with the evidence.
Naturally people will try to fabricate evidence, and even good faith evidence may be unreliable, so you'll have to do your best to access it's veracity. But what the adversary believes or appears to believe is largely immaterial.
This feels like the same kind of vague "rational mysticism." "We don't know what we don't know, and we're such silly humans, therefore...AI will kill us all" is all I can really take from it.
I really seriously doubt that an AI can convince a normal person to let it out, if they know they'll have their pay docked and if the communication is over text. The best scammers in the world can't convince most people to click on a link over text, let alone if "not clicking any links" was someone's job title.
- Oscar Wilde
When you come across an adversary, it’s to your benefit to try to bring them to the healthy side of things.
People can be pretty reasonable, and if they’re not then they can be shamed into behaving. If they cannot be shamed, then there’s retribution. If that doesn’t work, then there’s always the option to go full Rambo.
You never want to go full Rambo, but your adversary must understand that it’s an option that’s available to you. I don’t think super AI will be any different as an adversary, but maybe there’s something I haven’t considered.
What would a task force built to oppose the IRA seeding discord online look like? How would it operate? We need that.
The white-anting by Russia hasn't really triggered this kind of "immune response" - it's hard to know what to do about it, which is of course the entire point.
If you know more than others, that's great, but in that case please share some of what you know, so the rest of us can learn.
Cheaters in games like Magic are very rare; if most people tried to cheat whenever they thought they could get away with it, we'd be forced to set up competitions with more stringent verifiable rules like (off the top of my head) "all cards must be drawn and given to you from your deck by the opponent". We haven't done that, so I infer that most people don't try to cheat.
(In your favour, I do concede that everyone writes down their individual understanding of the history of a given chess game; but there are weak instrumental reasons for that even if people aren't cheating, because it is possible to upset a board by accident.)
The number of people who declare they can totally trust what an adversary says because they agree with it is astounding, as though a committed opponent wouldn't do anything if it gained advantage including feinting in a way which seems unadvantegeous to gain long term advantage.
In the end, the only winning move is not to play. If you believe an adversary said something (or "a liar" if you prefer), you ignore it entirely. You make your mind up about what you believe based on evidence, and you decide if you agree with someone based on how well their statement comports with the evidence.
Naturally people will try to fabricate evidence, and even good faith evidence may be unreliable, so you'll have to do your best to access it's veracity. But what the adversary believes or appears to believe is largely immaterial.