Why are there so many rationalist cults?

I don’t think it’s just (or even particularly) bad axioms, I think it’s that people tend to build up “logical” conclusions where they think each step is a watertight necessity that follows inevitably from its antecedents, but actually each step is a little bit leaky, leading to runaway growth in false confidence.

Not that non-rationalists are any better at reasoning, but non-rationalists do at least benefit from some intellectual humility.

> I immediately become suspicious of anyone who is very certain of something

Me too, in almost every area of life. There's a reason it's called a conman: they are tricking your natural sense that confidence is connected to correctness.

But also, even when it isn't about conning you, how do people become certain of something? They ignored the evidence against whatever they are certain of.

People who actually know what they're talking about will always restrict the context and hedge their bets. Their explanation are tentative, filled with ifs and buts. They rarely say anything sweeping.

I find Yudowsky-style rationalists morbidly fascinating in the same way as Scientologists and other cults. Probably because they seem to genuinely believe they're living in a sci-fi story. I read a lot of their stuff, probably too much, even though I find it mostly ridiculous.

The biggest nonsense axiom I see in the AI-cult rationalist world is recursive self-improvement. It's the classic reason superintelligence takeoff happens in sci-fi: once AI reaches some threshold of intelligence, it's supposed to figure out how to edit its own mind, do that better and faster than humans, and exponentially leap into superintelligence. The entire "AI 2027" scenario is built on this assumption; it assumes that soon LLMs will gain the capability of assisting humans on AI research, and AI capabilities will explode from there.

But AI being capable of researching or improving itself is not obvious; there's so many assumptions built into it!

- What if "increasing intelligence", which is a very vague goal, has diminishing returns, making recursive self-improvement incredibly slow?

- Speaking of which, LLMs already seem to have hit a wall of diminishing returns; it seems unlikely they'll be able to assist cutting-edge AI research with anything other than boilerplate coding speed improvements.

- What if there are several paths to different kinds of intelligence with their own local maxima, in which the AI can easily get stuck after optimizing itself into the wrong type of intelligence?

- Once AI realizes it can edit itself to be more intelligent, it can also edit its own goals. Why wouldn't it wirehead itself? (short-circuit its reward pathway so it always feels like it's accomplished its goal)

Knowing Yudowsky I'm sure there's a long blog post somewhere where all of these are addressed with several million rambling words of theory, but I don't think any amount of doing philosophy in a vacuum without concrete evidence could convince me that fast-takeoff superintelligence is possible.

One of the only idioms that I don't mind living my life by is, "Follow the truth-seeker, but beware those who've found it".

> I immediately become suspicious of anyone who is very certain of something, especially if they derived it on their own.

Are you certain about this?

There should be an extremist cult of people who are certain only that uncertainty is the only certain thing

They mostly seem to lean that way because it gives them carte blanche to do as they please. It is just a modern version of 'god has led my hand'.

Humans are not chickens, but we sure do seem to love having a pecking order.

To be clear, this article isn't calling rationalism a cult, it's about cults that have some sort of association with rationalism (social connection and/or ideology derived from rationalist concepts), e.g. the Zizians.

Yeah, a lot of the comments here are really just addressing cults writ large and opposed to why this one was particularly successful.

A significant part of this is the intersection of the cult with money and status - this stuff really took off once prominent SV personalities became associated with it, and got turbocharged when it started intersecting with the angel/incubator/VC scene, when there was implicit money involved.

It's unusually successful because -- for a time at least -- there was status (and maybe money) in carrying water for it.

I've always been under the impression that M:tA's rules of How Magic Works are inspired by actual mystical beliefs that people have practiced for centuries. It's probably about as much of a magical for mystical development as the GURPS Cyberpunk rulebook was for cybercrime but it's pointing at something that already exists and saying "this is a thing we are going to tell an exaggerated story about".

See for example "Reality Distortion Field": https://en.wikipedia.org/wiki/Reality_distortion_field

Came to ask a similar question, but also has it always been like this? The difference is now these people/groups on the fringe had no visibility before the internet?

It's nuts.

I don't know how you can call yourself a "rationalist" and base your worldview on a fantasy game.

It's been like this a while. Have you heard the tale of the Final Fantasy House?: http://www.demon-sushi.com/warning/

Cult leaders tend to be narcissists.

Narcissists tend to believe that they are always right, no mater what the topic is, or how knowledgeable they are. This makes them speak with confidence and conviction.

Some people are very drawn to confident people.

If the cult leader has other mental health issues, it can/will seep into their rhetoric. Combine that with unwavering support from loyal followers that will take everything they say as gospel...

That's about it.

> How did we get to this place with people going completely nuts like this?

Ayahuasca?

I'm entertaining sending my kiddo to a Waldorf School, because it genuinely seems pretty good.

But looking into the underlying Western Esoteric Spirit Science, 'Anthroposophy' (because Theosophy wouldn't let him get weird enough) by Rudolph Steiner, has been quite a ride. The point being that.. humans have a pretty endless capacity to go ALL IN on REALLY WEIRD shit, as long as it promises to fix their lives if they do everything they're told. Naturally if their lives aren't fixed, then they did it wrong or have karmic debt to pay down, so YMMV.

In any case, I'm considering the latent woo-cult atmosphere as a test of the skeptical inoculation that I've tried to raise my child with.

I came to comments first. Thank you for sharing this quote. Gave me a solid chuckle.

I think people are going nuts because we've drifted from the dock of a stable civilization. Institutions are a mess. Economy is a mess. Combine all of that together with the advent of social media making the creation of echo chambers (and the inevitable narcissism of "leaders" in those echo chambers) effortless and ~15 years later, we have this.

From false premises, you can logically and rationally reach really wrong conclusions. If you have too much pride in your rationality, you may not be willing to say "I seem to have reached a really insane conclusion, maybe my premises are wrong". That is, the more you pride yourself on your rationalism, the more prone you may be to accepting a bogus conclusion if it is bogus because the premises are wrong.

Makes me think of that saying that great artists steal, so repurposed for cult founders: "Good cult founders copy, great cult founders steal"

I do not think this cult dogma is any more out there than other cult dogma I have heard, but the above quote makes me think it is easier to found cults in modern day in someways since you can steal other complex world building from numerous sources rather building yourself and keeping everything straight.

Running a cult is a somewhat reliable source of narcissistic supply. The internet tells you how to do it. So an increasing number of people do it.

People are wired to worship, and want somebody in charge telling them what to do.

I'm a staunch atheist and I feel the pull all the time.

astronauts_meme.jpg

Who the fuck bases a Black Lotus cult on Mage: the Ascension rather than Magic: the Gathering? Is this just a mistake by the journalist?

I've met a fair share of people in the burner community, the vast majority I met are lovely folks who really enjoy the process of bringing some weird big idea into reality, working hard on the builds, learning stuff, and having a good time with others for months to showcase their creations at some event.

On the other hand, there's a whole other side of a few nutjobs who really behave like cult leaders, they believe their own bullshit and over time manage to find in this community a lot of "followers", since one of the foundational aspects is radical acceptance it becomes very easy to be nutty and not questioned (unless you do something egregious).

Humans are compelled to find agency and narrative in chaos. Evolution favored those who assumed the rustle was a predator, not the wind. In a post-Enlightenment world where traditional religion often fails (or is rejected), this drive doesn't vanish. We don't stop seeking meaning. We seek new frameworks. Our survival depended on group cohesion. Ostracism meant death. Cults exploit this primal terror. Burning Man's temporary city intensifies this: extreme environment, sensory overload, forced vulnerability. A camp like Black Lotus offers immediate, intense belonging. A tribe with shared secrets (the "Ascension" framework), rituals, and an "us vs. the sleepers" mentality. This isn't just social; it's neurochemical. Oxytocin (bonding) and cortisol (stress from the environment) flood the system, creating powerful, addictive bonds that override critical thought.

Human brains are lazy Bayesian engines. In uncertainty, we grasp for simple, all-explaining models (heuristics). Mage provides this: a complete ontology where magic equals psychology/quantum woo, reality is malleable, and the camp leaders are the enlightened "tradition." This offers relief from the exhausting ambiguity of real life. Dill didn't invent this; he plugged into the ancient human craving for a map that makes the world feel navigable and controllable. The "rationalist" veneer is pure camouflage. It feels like critical thinking but is actually pseudo-intellectual cargo culting. This isn't Burning Man's fault. It's the latest step of a 2,500-year-old playbook. The Gnostics and the Hermeticists provided ancient frameworks where secret knowledge ("gnosis") granted power over reality, accessible only through a guru. Mage directly borrows from this lineage (The Technocracy, The Traditions). Dill positioned himself as the modern "Ascended Master" dispensing this gnosis.

The 20th century cults Synanon, EST, Moonies, NXIVM all followed similar patterns, starting with isolation. Burning Man's temporary city is the perfect isolation chamber. It's physically remote, temporally bounded (a "liminal space"), fostering dependence on the camp. Initial overwhelming acceptance and belonging (the "Burning Man hug"), then slowly increasing demands (time, money, emotional disclosure, sexual access), framed as "spiritual growth" or "breaking through barriers" (directly lifted from Mage's "Paradigm Shifts" and "Quintessence"). Control language ("sleeper," "muggle," "Awakened"), redefining reality ("that rape wasn't really rape, it was a necessary 'Paradox' to break your illusions"), demanding confession of "sins" (past traumas, doubts), creating dependency on the leader for "truth."

Burning Man attracts people seeking transformation, often carrying unresolved pain. Cults prey on this vulnerability. Dill allegedly targeted individuals with trauma histories. Trauma creates cognitive dissonance and a desperate need for resolution. The cult's narrative (Mage's framework + Dill's interpretation) offers a simple explanation for their pain ("you're unAwakened," "you have Paradox blocking you") and a path out ("submit to me, undergo these rituals"). This isn't therapy; it's trauma bonding weaponized. The alleged rape wasn't an aberration; it was likely part of the control mechanism. It's a "shock" to induce dependency and reframe the victim's reality ("this pain is necessary enlightenment"). People are adrift in ontological insecurity (fear about the fundamental nature of reality and self). Mage offers a new grand narrative with clear heroes (Awakened), villains (sleepers, Technocracy), and a path (Ascension).

Mage: The Ascension is basically a delusions of grandeur simulator, so I can see how an already unstable personality might get attached to it and become more unstable.

Paraphrasing someone I don't recall - when people believe in nothing, they'll believe anything.

Capital-R Rationalism also encourages you to think you can outperform it, by being smart and reasoning from first principles. That was the idea behind MetaMed, founded by LessWronger Michael Vassar - that being trained in rationalism made you better at medical research and consulting than medical school or clinical experience. Fortunately they went out of business before racking up a body count.

One lesson I've learned and seen a lot in my life is that understanding that something is wrong or what's wrong about it, and being able to come up with a better solution are distinct, and the latter is often much harder. It seems often that those that are best able to describe the problem often don't overlap much with those that can figure out how to solve, even though they think they can.

It is an unfortunate reality of our existence that sometimes Chesterton actually did build that fence for a good reason, a good reason that's still here.

(One of my favorite TED talks was about a failed experiment in introducing traditional Western agriculture to a people in Zambia. It turns out when you concentrate too much food in one place, the hippos come and eat it all and people can't actually out-fight hippos in large numbers. In hindsight, the people running the program should have asked how likely it was that folks in a region that had exposure to other people's agriculture for thousands of years, hadn't ever, you know... tried it. https://www.ted.com/talks/ernesto_sirolli_want_to_help_someo...)

It's almost the defining characteristic of our time.

Sure, but that doesn't say anything about why one particular social scene would spawn a bunch of cults while others do not, which is the question that the article is trying to answer.

Yeah, I think this is exactly it. If something sounds extremely stupid, or if everyone around you says it's extremely stupid, it probably is. If you can't justify it, it's probably because you have failed to find the reason it's stupid, not because it's actually genius.

And the crazy thing is, none of that is fundamentally opposed to rationalism. You can be a rationalist who ascribes value to gut instinct and societal norms. Those are the product of millions of years of pre-training.

I have spent a fair bit of time thinking about the meaning of life. And my conclusions have been pretty crazy. But they sound insane, so until I figure out why they sound insane, I'm not acting on those conclusions. And I'm definitely not surrounding myself with people who take those conclusions seriously.

> The whole game of Rationalism is that you should ignore gut intuitions and cultural norms that you can't justify with rational arguments.

Specifically, rationalism spends a lot of time about priors, but a sneaky thing happens that I call the 'double update'.

Bayesian updating works when you update your genuine prior believe with new evidence. No one disagrees with this, and sometimes it's easy and sometimes it's difficult to do.

What Rationalists often end up doing is relaxing their priors - intuition, personal experience, cultural norms - and then updating. They often think of this as one update, but what it is is two. The first update, relaxing priors, isn't associated with evidence. It's part of the community norms. There is an implicit belief that by relaxing one's priors you're more open to reality. The real result though, is that it sends people wildly off course. Care in point: all the cults.

Consider the pre-tipped scale. You suspect the scale reads a little low, so before weighing you tilt it slightly to "correct" for that bias. Then you pour in flour until the dial says you've hit the target weight. You’ve followed the numbers exactly, but because you started from a tipped scale, you've ended up with twice the flour the recipe called for.

Trying to correct for bias by relaxing priors is updating using evidence, not just because everyone is doing it.

> The whole game of Rationalism is that you should ignore gut intuitions and cultural norms that you can't justify with rational arguments.

The game as it is _actually_ played is that you use rationalist arguments to justify your pre-existing gut intuitions and personal biases.

Even the real progenitors of a lot of this sort of thought, like E.T. Jaynes, expoused significantly more skepticism than I've ever seen a "rationalist" use. I would even imagine if you asked almost all rationalists who E.T. Jaynes was (if they weren't well versed in statistical mechanics) they'd have no idea who he was or why his work was important to applying "Bayesianism".

I'm lucky enough work in a pretty rational place (small "r"). We're normally data-limited. Being "more rational" would mean taking/finding more of the right data, talking to the right people, reading the right stuff. Not just thinking harder and harder about what we already know.

There's a point where more passive thinking stops adding value and starts subtracting sanity. It's pretty easy to get to that point. We've all done it.

I don't disagree, but to steelman the case for (neo)rationalism: one of its fundamental contributions is that Bayes' theorem is extraordinarily important as a guide to reality, perhaps at the same level as the second law of thermodynamics; and that it is dramatically undervalued by larger society. I think that is all basically correct.

(I call it neorationalism because it is philosophically unrelated to the more traditional rationalism of Spinoza and Descartes.)

People find academic philosophy impenetrable and pretentious, but it has two major advantages over rationalist cargo cults.

The first is diffusion of power. Social media is powered by charisma, and while it is certainly true that personality-based cults are nothing new, the internet makes it way easier to form one. Contrast that with academic philosophy. People can have their own little fiefdoms, and there is certainly abuse of power, but rarely concentrated in such a way that you see within rationalist communities.

The second (and more idealistic) is that the discipline of Philosophy is rooted in the Platonic/Socratic notion that "I know that I know nothing." People in academic philosophy are on the whole happy to provide a gloss on a gloss on some important thinker, or some kind of incremental improvement over somebody else's theory. This makes it extremely boring, and yet, not nearly as susceptible to delusions of grandeur. True skepticism has to start with questioning one's self, but everybody seems to skip that part and go right to questioning everybody else.

Rationalists have basically reinvented academic philosophy from the ground up with none of the rigor, self-discipline, or joy. They mostly seem to dedicate their time to providing post-hoc justifications for the most banal unquestioned assumptions of their subset of contemporary society.

Yeah, I would say the groups in question are notionally, aspirationally rational and I would hate for the takeaway to be disengagement from principles of critical thinking and skeptical thinking writ large.

Which, to me, raises the fascinating question of what does a "good" version look like, of groups and group dynamics centered around a shared interest in best practices associated with critical thinking?

At a first impression, I think maybe these virtues (which are real!) disappear into the background of other, more applied specializations, whether professions, hobbies, backyard family barbecues.

The article is talking about cults that arose out of the rationalist social milieu, which is a separate question from whether the cult's beliefs qualify as "rationalist" in some sense (a question that usually has no objective answer anyway).

>so world models including demons have never predicted any of their sense data.

There's a reason they call themselves "rationalists" instead of empiricists or positivists. They perfectly inverted Hume ("reason is, and ought only to be the slave of the passions")

These kinds of harebrained views aren't an accident but a product of rationalism. The idea that intellect is quasi infinite and that the world can be mirrored in the mind is not running contradictory to, but just the most extreme form of rationalism taken to its conclusion, and of course deeply religious, hence the constant fantasies about AI divinities and singularities.

> for whatever reason are no longer affiliated with the church of their youth.

This is the Internet, you're allowed to say "they are obsessed with unlimited drugs and weird sex things, far beyond what even the generally liberal society tolerates".

I'm increasingly convinced that every other part of "Rationalism" is just distraction or justification for those; certainly there's a conscious decision to minimize talking about this part on the Internet.

> Obviously the article intends to make the case that this is a cult

The author is a self-identified rationalist. This is explicitly established in the second sentence of the article. Given that, why in the world would you think they're trying to claim the whole movement is a cult?

Obviously you and I have very different definitions of "obvious"

In fact, I'd go a step further and note the similarity with organized religion. People have a tendency to organize and dogmatize everything. The problem with religion is rarely the core ideas, but always the desire to use it as a basis for authority, to turn it dogmatic and ultimately form a power structure.

And I say this as a Christian. I often think that becoming a state religion was the worst thing that ever happened to Christianity, or any religion, because then it unavoidably becomes a tool for power and authority.

And doing the same with other ideas or ideologies is no different. Look at what happened to communism, capitalism, or almost any other secular idea you can think of: the moment it becomes established, accepted, and official, the corruption sets in.

I do not see any reasons for you to get down-voted.

I agree that the term "rationalist" would appeal to many people, and the obvious need to belong to a group plays a huge role.

How has he fared in the fields of philosophy and AI research in terms of peer review, is there some kind of roundup or survey around about this?

Don't forget the biggest scifi guy turned cult leader of all L. Ron Hubbard

Boring as it is, this is the answer. It's just more religion.

  Church, cult, cult, church. So we'll get bored someplace else every Sunday. Does this really change our everyday lives?

IIUC the name in its current sense was sort of an accident. Yudkowsky originally used the term to mean "someone who succeeds at thinking and acting rationally" (so "correctism" or "winsargumentism" would have been about equally good), and then talked about the idea of "aspiring rationalists" as a community narrowly focused on developing a sort of engineering discipline that would study the scientific principles of how to be right in full generality and put them into practice. Then the community grew and mutated into a broader social milieu that was only sort of about that, and people needed a name for it, and "rationalists" was already there, so that became the name through common usage. It definitely has certain awkwardnesses.

To be honest I don't understand that objection. If you strip it from all its culty sociological effects, one of the original ideas of rationalism was to try to use logical reasoning and statistical techniques to explicitly avoid the pitfalls of known cognitive biases. Given that foundational tenet, "rationalism" seems like an extremely appropriate moniker.

I fully accept that the rationalist community may have morphed into something far beyond that original tenet, but I think rationalism just describes the approach, not that it's the "one true philosophy".

Objectivisim?

Great names! Are you using them, or are they available? /s

None of them are practicing Buddhist meditation though, same for the "personal growth" oriented meditation styles.

Buddhist meditation exists only in the context of the Four Noble Truths and the rest of the Buddha's Dhamma. Throwing them away means it stops being Buddhist.

I was wondering about this too. You could also say it's a sturgeon's law question.

They do note at the beginning of the article that many, if not most such groups have reasonably normal dynamics, for what it's worth. But I think there's a legitimate question of whether we ought to expect groups centered on rational thinking to be better able to escape group dynamics we associate with irrationality.

Rationalists, in this case, refers specifically to the community clustered around LessWrong, which explicitly and repeatedly emphasizes points like "you can't claim to have a well grounded belief if you don't actually have empirical evidence for it" (https://www.lesswrong.com/w/evidence for a quick overview of some of the basic posts on that topic)

To quote one of the core foundational articles: "Before you try mapping an unseen territory, pour some water into a cup at room temperature and wait until it spontaneously freezes before proceeding. That way you can be sure the general trick—ignoring infinitesimally tiny probabilities of success—is working properly." (https://www.lesswrong.com/posts/eY45uCCX7DdwJ4Jha/no-one-can...)

One can argue how well the community absorbs the lesson, but this certainly seems to be a much higher standard than average.

That is the definition of “rationalism” as proposed by philosophers like Descartes and Kant, but I don’t think that is an accurate representation of the type of “rationalism” this article describes.

This article describes “rationalism” as described in LessWrong and the sequences by Eliezer Yudkowsky. A good amount of it based on empirical findings from psychology behavior science. It’s called “rationalism” because it seeks to correct common reasoning heuristics that are purported to lead to incorrect reasoning, not in contrast to empiricism.

But you cannot have reason without substantial proof of how things behave by observing them in the first place. Reason is simply a logical approach to yes and no questions where you factually know, from observation of past events, how things work. And therefore you can simulate an outcome by the exercise of reasoning applied onto a situation that you have not yet observed and come to a logical outcome, given the set of rules and presumptions.

Depends very much on what you're hoping to get out of it. There isn't really one "rationalist" thing at this point, it's now a whole bunch of adjacent social groups with overlapping-but-distinct goals and interests.

https://www.lesswrong.com/highlights this is the ostensible "Core Highlights", curated by major members of the community, and I believe Eliezer would endorse it.

If you don't get anything out of reading the list itself, then you're probably not going to get anything out of the rest of the community either.

If you poke around and find a few neat ideas there, you'll probably find a few other neat ideas.

For some people, though, this is "wait, holy shit, you can just DO that? And it WORKS?", in which case probably read all of this but then also go find a few other sources to counter-balance it.

(In particular, probably 90% of the useful insights already exist elsewhere in philosophy, and often more rigorously discussed - LessWrong will teach you the skeleton, the general sense of "what rationality can do", but you need to go elsewhere if you want to actually build up the muscles)

Because we are currently living in an age of narcissism and tribalism / Identitarianism is the societal version of narcissism.

The question the article is asking is "why did so many cults come out of this particular social milieu", not "why are there a lot of cults in the whole world".

Your profile says that you want to keep your identity small, but you have like over 30 thousand comments spelling out exactly who you are and how you think. Why not shard accounts? Anyways. Just a random thought.

The book Imagined Communities (Benedict Anderson) touches on this, making the case that in modern times, "nation" has replaced the cultural narrative purpose previously held by "tribe," "village," "royal subject," or "religion."

The shared thread among these is (in ever widening circles) a story people tell themselves to justify precisely why, for example, the actions of someone you'll never meet in Tulsa, OK have any bearing whatsoever on the fate of you, a person in Lincoln, NE.

One can see how this leaves an individual in a tenuous place if one doesn't feel particularly connected to nationhood (one can also see how being too connected to nationhood, in an exclusionary way, can also have deleterious consequences, and how not unlike differing forms of Christianity, differing concepts on what the 'soul' of a nation is can foment internal strife).

(To be clear: those fates are intertwined to some extent; the world we live in grows ever smaller due to the power of up-scaled influence of action granted by technology. But "nation" is a sort of fiction we tell ourselves to fit all that complexity into the slippery meat between human ears).

Also, who would want to join an "irrationalist cult" ?

While deployment of logical fallacies to win arguments is annoying at best, the far bigger problem is that people make those fallacies in the first place — such as not considering base rates.

It's generally worth remembering that some of the fallacies are actually structural, and some are rhetorical.

A contradiction creates a structural fallacy; if you find one, it's a fair belief that at least one of the supporting claims is false. In contrast, appeal to authority is probabilistic: we don't know, given the current context, if the authority is right, so they might be wrong... But we don't have time to read the universe into this situation so an appeal to authority is better than nothing.

... and this observation should be coupled with the observation that the school of rhetoric wasn't teaching a method for finding truth; it was teaching a method for beating an opponent in a legal argument. "Appeal to authority is a logical fallacy" is a great sword to bring to bear if your goal is to turn off the audience's ability to ask whether we should give the word of the environmental scientist and the washed-up TV actor equal weight on the topic of environmental science...

There's nothing irrational about it, this is how you maximize power and profit at any and all costs.

Scientology is here since 1953 and it has similarly bonkers set of believes. And is huge.

Your rant about government or not being allowed to hike in some places in Canada is unrelated to the issue.