This paper presents a theoretical proof that AGI systems will structurally collapse under certain semantic conditions — not due to lack of compute, but because of how entropy behaves in heavy-tailed decision spaces.
The idea is called IOpenER: Information Opens, Entropy Rises. It builds on Shannon’s information theory to show that in specific problem classes (those with α ≤ 1), adding information doesn’t reduce uncertainty — it increases it. The system can’t converge, because meaning itself keeps multiplying.
The core concept — entropy divergence in these spaces — was already present in my earlier paper, uploaded to PhilArchive on June 1. This version formalizes it. Apple’s study, The Illusion of Thinking, was published a few days later. It shows that frontier reasoning models like Claude 3.7 and DeepSeek-R1 break down exactly when problem complexity increases — despite adequate inference budget.
I didn’t write this paper in response to Apple’s work. But the alignment is striking. Their empirical findings seem to match what IOpenER predicts.
Curious what this community thinks: is this a meaningful convergence, or just an interesting coincidence?
Thanks for this - Looking forward to reading the full paper.
That said, the most obvious objection that comes to mind about the title is that … well, I feel that I’m generally intelligent, and therefore general intelligence of some sort is clearly not impossible.
Can you give a short précis as to how you are distinguishing humans and the “A” in artificial?
ICBTheory · 2h ago
Sure I can (and thanks for writing)
Well, given the specific way you asked that question I confirm your self assertion - and am quite certain that your level of Artificiality converges to zero, which would make you a GI without A...
- You stated to "feel" generally intelligent (A's don't feel and don't have an "I" that can feel)
- Your nuanced, subtly ironic and self referential way of formulating clearly suggests that you are not a purely algorithmic entity
A "précis" as you wished:
Artificial — in the sense used here (apart from the usual "planfully built/programmed system" etc.) — algorithmic, formal, symbol-bound.
Humans as "cognitive system" have some similar traits of course - but obviously, there seems to be more than that.
rusk · 3h ago
Not the person asked, but in time honoured tradition I will venture forth that the key difference is billions of years of evolution. Innumerable blooms and culls. And a system that is vertically integrated to its core and self sustaining.
ben_w · 3h ago
The mathematical proof, as you describe it, sounds like the "No Free Lunch theorem". Humans also can't generalise to learning such things.
As you note in 2.1, there is widespread disagreement on what "AGI" means. I note that you list several definitions which are essentially "is human equivalent". As humans can be reduced to physics, and physics can be expressed as a computer program, obviously any such definition can be achieved by a sufficiently powerful computer.
For 3.1, you assert:
"""
Now, let's observe what happens when an Al system - equipped with state-of-the-art natural language processing, sentiment analysis, and social reasoning - attempts to navigate this question.
The Al begins its analysis:
• Option 1: Truthful response based on biometric data → Calculates likely negative emotional impact → Adjusts for honesty parameter → But wait, what about relationship history? → Recalculating...
• Option 2: Diplomatic deflection → Analyzing 10,000 successful deflection patterns → But tone matters → Analyzing micro-expressions needed → But timing matters → But past conversations matter → Still calculating...
• Option 3: Affectionate redirect → Processing optimal sentiment → But what IS optimal here? The goal keeps shifting → Is it honesty? Harmony? Trust? → Parameters unstable → Still calculating...
• Option n: ....
Strange, isn't it? The Al hasn't crashed. It's still running. In fact, it's generating more and more nuanced analyses. Each additional factor may open ten new considerations. It's not getting closer to an answer - it's diverging.
"""
Which AI? ChatGPT just gives an answer. Your other supposed examples have similar issues in that it looks like you've *imagined* an AI rather than having tried asking an AI to seeing what it actually does or doesn't do.
I'm not reading 47 pages to check for other similar issues.
WhitneyLand · 3h ago
“This paper presents a theoretical proof that AGI systems will structurally collapse under certain semantic conditions…”
No it doesn’t.
Shannon entropy measures statistical uncertainty in data. It says nothing about whether an agent can invent new conceptual frames. Equating “frame changes” with rising entropy is a metaphor, not a theorem, so it doesn’t even make sense as a mathematical proof.
This is philosophical musing at best.
ICBTheory · 1h ago
Correct: Shannon entropy originally measures statistical uncertainty over a fixed symbol space. When the system is fed additional information/data, then entropy goes down, uncertainty falls. This is always true in situations where the possible outcomes are a) sufficiently limited and b)unequally distributed. In such cases, with enough input, the system can collapse the uncertainty function within a finite number of steps.
But the paper doesn’t just restate Shannon.
It extends this very formalism to semantic spaces where the symbol set itself becomes unstable.
These situations arise when (a) entropy is calculated across interpretive layers (as in LLMs), and (b) the probability distribution follows a heavy-tailed regime (α ≤ 1).
Under these conditions, entropy divergence becomes mathematically provable.
This is far from being metaphorical: it’s backed by formal Coq-style proofs (see Appendix C in he paper).
AND: it is exactly the mechanism that can explain the Apple-Papers' results
yodon · 2h ago
I'm wondering if you may have rediscovered the concept of "Wicked Problems", which have been studied in system analysis and sociology since the 1970's (I'd cite the Wikipedia page, but I've never been particularly fond of Wikipedia's write up on them). They may be worth reading up on if you're not familiar with them.
gremlinsinc · 2h ago
does this include if the AI can devise new components and use drones and things essentially to build a new iteration of itself more capable to compute a thing and keep repeating this going out into the universe as needed for resources and using von Neumann probes.. etc?
proc0 · 3h ago
The paper is skipping over the definition of AI. It jumps right into AGI, and that depends on what AI means. It could be LLMs, deep neural networks, or any possible implementation on a Turing machine. The latter I suspect would be extremely difficult to prove. So far almost everything can be simulated by Turing machines and there's no reason it couldn't also simulate human brains, and therefore AGI. Even if the claim is that human brains are not enough for GI (and that our bodies are also part of the intelligence equation), we could still simulate an entire human being down to every cell, in theory (although in practice it wouldn't happen anytime soon, unless maybe quantum computers, but I digress).
Still an interesting take and will need to dive in more, but already if we assume the brain is doing information processing then the immediate question is how can the brain avoid this problem, as others are pointing out. Is biological computation/intelligence special?
Takashoo · 1h ago
Turing machines only model computation. Real life is interaction. Check the work of Peter Wegner. When interaction machines enter into the picture, AI can be embodied, situated and participate in adaptation processes. The emergent behaviour may bring AGI in a pragmatic perspective. But interaction is far more expressive than computation rendering theoretical analysis challenging.
proc0 · 1h ago
Interaction is just another computation, and clearly we can interact with computers, and also simulate that interaction within the computer, so yes Turing machines can handle it. I'll check out Wegner.
cainxinth · 29m ago
The crux here is the definition of AGI. The author seems to say that only an endgame, perfect information processing system is AGI. But that definition is too strict because we might develop something that is very far from perfect but which still feels enough like AGI to call it that.
viralsink · 2h ago
If I understood correctly, this is about finding solutions to problems that have an infinite solution space, where new information does not constrain it.
Humans don't have the processing power to traverse such vast spaces. We use heuristics, in the same way a chess player does not iterate over all possible moves.
It's a valid point to make, however I'd say this just points to any AGI-like system having the same epistemological issues as humans, and there's no way around it because of the nature of information.
Stephen Wolfram's computational irreducibility is another one of the issues any self-guided, phyiscally grounded computing engine must have. There are problems that need to be calculated whole. Thinking long and hard about possible end-states won't help. So one would rather have 10000 AGIs doing somewhat similar random search in the hopes that one finds something useful.
I guess this is what we do in global-scale scientific research.
Animats · 3h ago
Penrose did this argument better.[1] Penrose has been making that argument for thirty years, and it played better before AI started getting good.
AI via LLMs has limitations, but they don't come from computability.
Thanks — and yes, Penrose’s argument is well known.
But this isn’t that, as I’m not making a claim about consciousness or invoking quantum physics or microtubules (which, I agree, are highly speculative).
The core of my argument is based on computability and information theory — not biology.
Specifically: that algorithmic systems hit hard formal limits in decision contexts with irreducible complexity or semantic divergence, and those limits are provable using existing mathematical tools (Shannon, Rice, etc.).
So in some way, this is the non-microtubule version of AI critique. I don’t have the physics background to engage in Nobel-level quantum speculation — and, luckily, it’s not needed here.
daedrdev · 3h ago
Clearly nature avoids this problem. So theoretically by replicating natural selection or something else in AI models, which arguably we already do, the theoretical entropy trap clearly can be avoided, we aren't even potentially decreasing entropy with AI training since doing so uses power generation which increases entropy
rusk · 3h ago
It can be avoided certainly, but can it be avoided with the current or near term technology about which many are saying “it’s only a matter of time”
kelseyfrog · 1h ago
> And - as wonderfully remarkable as such a system might be - it would, for our investigation, be neither appropriate nor fair to overburden AGI by an operational definition whose implicit metaphysics and its latent ontological worldviews lead to the epistemology of what we might call a “total isomorphic a priori” that produces an algorithmic world-formula that is identical with the world itself (which would then make the world an ontological algorithm...?).
> Anyway, this is not part of the questions this paper seeks to answer. Neither will we wonder in what way it could make sense to measure the strength of a model by its ability to find its relative position to the object it models. Instead, we chose to stay ignorant - or agnostic? - and take this fallible system called "human". As a point of reference.
Cowards.
That's the main counter argument and acknowledging its existence without addressing it is a craven dodge.
Assuming the assumptions[1] are true, then human intelligence isn't even able to be formalized under the same pretext.
Either human intelligence isn't
1. Algorithmic. The main point of contention. If humans aren't algorithmically reducible - even at the level computation of physics, then human cognition is supernatural.
2. Autonomous. Trivially true given that humans are the baseline.
3. Comprehensive (general): Trivially true since humans are the baseline.
4. Competent: Trivially true given humans are the baseline.
I'm not sure how they reconcile this given that they simply dodge the consequences that it implies.
Overall, not a great paper. It's much more likely that their formalism is wrong than their conclusion.
Footnotes
1. not even the consequences, unfortunately for the authors.
ICBTheory · 19m ago
Just to make sure I understand:
–Are we treating an arbitrary ontological assertion as if it’s a formal argument that needs to be heroically refuted? Or better: is that metaphysical setup an argument?
If that’s the game, fine. Here we go:
– The claim that one can build a true, perfectly detailed, exact map of reality is… well... ambitious. It sits remarkably far from anything resembling science , since it’s conveniently untouched by that nitpicky empirical thing called evidence. But sure: freed from falsifiability, it can dream big and give birth to its omnicartographic offspring.
– oh, quick follow-up: does that “perfect map” include itself? If so... say hi to Alan Turing. If not... well, greetings to Herr Goedel.
– Also: if the world only shows itself through perception and cognition, how exactly do you map it “as it truly is”? What are you comparing your map to — other observations? Another map?
– How many properties, relations, transformations, and dimensions does the world have? Over time? Across domains? Under multiple perspectives? Go ahead, I’ll wait... (oh, and: hi too.. you know who)
And btw the true detailed map of the world exists.... It’s the world.
It’s just sort of hard to get a copy of it. Not enough material available ... and/or not enough compute....
P.S. Sorry if that came off sharp — bit of a spur-of-the-moment reply.
If you want to actually dig into this seriously, I’d be happy to.
tim333 · 2h ago
This sounds rather silly. Given the usual definition of AGI as being human like intelligence with some variation on how smart the humans are, and the fact that humans use a network of neurons that can largely be simulated by an artificial network of neurons, it's probably twaddle largely.
like_any_other · 3h ago
So does the human brain transcend math, or are humans not generally intelligent?
geoka9 · 2h ago
Humans are fallible in a way computers are not. One could argue any creative process is an exercise in fallibility.
More interestingly, humans are capable of assessing the results of their "neural misfires" ("hmm, there's something to this"), whereas even if we could make a computer do such mistakes, it wouldn't know its Penny Lane from its Daddy's Car[0], even if it managed to come up with one.
Hang on, hasn't everyone spent the past few years complaining about LLMs and diffusion models being very fallible?
And we can get LLMs to do better by just prompting them to "think step by step" or replacing the first ten attempts to output a "stop" symbolic token with the token for "Wait… "?
No comments yet
ICBTheory · 3h ago
Hi and thanks for engaging :-)
Well, it in fact depends on what intelligence is to your understanding:
-If it intelligence = IQ, i.e. the rational ability to infer, to detect/recognize and extrapolate patterns etc, then AI is or will soon be more intelligent than us, while we humans are just muddling through or simply lucky having found relativity theory and other innovations just at the convenient moment in time ... So then, AI will soon also stumble over all kind of innovations. None of both will be able to deliberately think beyond what is thinkable at the respective present.
- But If intelligence is not only a level of pure rational cognition, but maybe an ability to somehow overcome these frame-limits, then humans obviously exert some sort of abilities that are beyond rational inference. Abilities that algorithms can impossibly reach, as all they can is compute.
- Or: intelligence = IQ, but it turns out to be useless in big, pivotal situations where you’re supposed to choose the “best” option — yet the set of possible options isn’t finite, knowable, or probabilistically definable. There’s no way to defer to probability, to optimize, or even to define what “best” means in a stable way. The whole logic of decision collapses — and IQ has nothing left to grab onto.
The main point is: neither algorithms nor rationality can point beyond itself.
In other words: You cannot think out of the box - thinking IS the box.
(maybe have a quick look at my first proof -last chapter before conclusion- - you will find a historical timeline on that IQ-Thing)
like_any_other · 1h ago
Let me steal another users alternate phrasing: Since humans and computers are both bound by the same physical laws, why does your proof not apply to humans?
autobodie · 3h ago
Humans do a lot of things that computers don't, such as be born, age (verb), die, get hungry, fall in love, reproduce, and more. Computers can only metaphorically do these things, human learning is correlated with all of them, and we don't confidently know how. Have some humility.
onlyrealcuzzo · 3h ago
The point is that if it's mathematically possible for humans, than it naively would be possible for computers.
All of that just sounds hard, not mathematically impossible.
As I understand it, this is mostly a rehash on the dated Lucas Penrose argument, which most Mind Theory researches refute.
andyjohnson0 · 3h ago
TFA presents an information-theoretic argument forAGI being impossible. My reading of your parent commenter is that they are asking why this argument does not also apply to humans.
You make broadly valid points, particularly about the advantages of embodyment, but I just dont think theyre good responses to the theoretical article under discussion (or the comment that you were responding to).
daedrdev · 3h ago
Taking GLP-1 makes me question how much hunger is really me versus my hormones controlling me.
ninetyninenine · 3h ago
We don’t even know how LLMs work. But we do know the underlying mechanisms are governed by math because we have a theory of reality that governs things down to the atomic scale and humans and LLMs are made out of atoms.
So because of this we know reality is governed by maths. We just can’t fully model the high level consequence of emergent patterns due to the sheer complexity of trillions of interacting atoms.
So it’s not that there’s some mysterious supernatural thing we don’t understand. It’s purely a complexity problem in that we only don’t understand it because it’s too complex.
What does humility have to do with anything?
hnfong · 3h ago
> we have a theory of reality that governs things down to the atomic scale and humans and LLMs are made out of atoms.
> So because of this we know reality is governed by maths.
That's not really true. You have a theory, and let's presume so far it's consistent with observations. But it doesn't mean it's 100% correct, and doesn't mean at some point in the future you won't observe something that invalidates the theory. In short, you don't know whether the theory is absolutely true and you can never know.
Without an absolutely true theory, all you have is belief or speculation that reality is governed by maths.
> What does humility have to do with anything?
Not the GP but I think humility is kinda relevant here.
ninetyninenine · 1h ago
>That's not really true. You have a theory, and let's presume so far it's consistent with observations. But it doesn't mean it's 100% correct, and doesn't mean at some point in the future you won't observe something that invalidates the theory. In short, you don't know whether the theory is absolutely true and you can never know.
Let me repharse it. As far as we know all of reality is governed by the principles of logic and therefore math. This is the most likely possibility and we have based all of our technology and culture and science around this. It is the fundamental assumption humanity has made on reality. We cannot consistently demonstrate disproof against this assumption.
>Not the GP but I think humility is kinda relevant here.
How so? If I assume all of reality is governed by math, but you don't. How does that make me not humble but you humble? Seems personal.
bigyabai · 3h ago
> We don’t even know how LLMs work
Speak for yourself. LLMs are a feedforward algorithm inferring static weights to create a tokenized response string.
We can compare that pretty trivially to the dynamic relationship of neurons and synapses in the human brain. It's not similar, case closed. That's the extent of serious discussion that can be had comparing LLMs to human thought, with apologies to Chomsky et. al. It's like trying to find the anatomical differences between a medieval scribe and a fax machine.
hnfong · 3h ago
Pretty sure in most other contexts you wouldn't agree a medieval scribe knows how a fax machine works.
ben_w · 3h ago
> Speak for yourself. LLMs are a feedforward algorithm inferring static weights to create a tokenized response string.
If we're OK with descriptions so lossy that they fit in a sentence, we also understand the human brain:
A electrochemical network with external inputs and some feedback loops, pumping ions around to trigger voltage cascades to create muscle contractions as outputs.
bigyabai · 51m ago
Yes. As long as we're confident in our definitions, that makes the questions easy. Is that the same as a feedforward algorithm inferring static weights to create a tokenized response string? Do you necessarily need an electrochemical network with external stimuli and feedback to generate legible text?
No. The answer is already solved; AI is not a brain, we can prove this by characteristically defining them both and using heuristic reasoning.
ben_w · 18m ago
> The answer is already solved; AI is not a brain, we can prove this by characteristically defining them both and using heuristic reasoning.
That "can" should be "could", else it presumes too much.
For both human brains and surprisingly small ANNs, far smaller than LLMs, humanity collectively does not yet know the defining characteristics of the aspects we care about.
I mean, humanity don't agree with itself what any of the three initials of AGI mean, there's 40 definitions of the word "consciousness", there are arguments about if there is either exactly one or many independent G-factors in human IQ scores, and also if those scores mean anything beyond correlating with school grades, and human nerodivergence covers various real states of existance that many of us find incomprehensible (sonetimes mutually, see e.g. most discussions where aphantasia comes up).
The main reason I expect little from an AI is that we don't know what we're doing. The main reason I can't just assume the least is because neither did evolution when we popped out.
ninetyninenine · 1h ago
George Hinton the person largely responsible about the AI revolution has this to say:
In that video above George Hinton, directly says we don't understand how it works.
So I don't speak just for myself. I speak for the person who ushered in the AI revolution, I speak for Experts in the field who know what they're talking aboutt. I don't speak for people who don't know what they're talking about.
Even though we know it's a feedforward network and we know how the queries are tokenized you cannot tell me what an LLM would say nor tell me why an LLM said something for a given prompt showing that we can't fully control an LLM because we don't fully understand it.
Don't try to just argue with me. Argue with the experts. Argue with the people who know more than you, Hinton.
bigyabai · 27m ago
Hinton invented the neural network, which is not the same as the transformer architecture used in LLMs. Asking him about LLM architectures is like asking Henry Ford if he can build a car from a bunch of scrap metal; of course he can't. He might understand the engine or the bodywork, but it's not his job to know the whole process. Nor is it Hinton's.
And that's okay - his humility isn't holding anyone back here. I'm not claiming to have memorized every model weight ever published, either. But saying that we don't know how AI works is empirically false; AI genuinely wouldn't exist if we weren't able to interpret and improve upon the transformer architecture. Your statement here is a dangerous extrapolation.
> you cannot tell me what an LLM would say nor tell me why an LLM said something for a given prompt showing that we can't fully control an LLM because we don't fully understand it.
You'd think this, but it's actually wrong. If you remove all of the seeded RNG during inference (meaning; no random seeds, no temps, just weights/tokenizer), you can actually create an equation that deterministically gives you the same string of text every time. It's a lot of math, but it's wholly possible to compute exactly what AI would say ahead of time if you can solve for the non-deterministic seeded entropy, or remove it entirely.
LLM weights and tokenizer are both always idempotent, the inference software often introduces variability for more varied responses. Just so we're on the same page here.
IAmGraydon · 3h ago
>We don’t even know how LLMs work.
Care to elaborate? Because that is utter nonsense.
Workaccount2 · 2h ago
We understand and build the trellis that the LLMs "grow" on. We don't have good insight into how a fully grown LLM actually turns any specific input into any specific output. We can follow it through the network, but it's a totally senseless noisy mess.
"Cat" lights up a certain set of neurons, but then "cat" looks completely different. That is what we don't really understand.
(This is an illustrative example made for easy understanding, not something I specifically went and compared)
EPWN3D · 2h ago
We don't know the path for how a given input produces a given output, but that doesn't mean we don't know how LLMs work.
We don't and can't know with certainty which specific atoms will fission in a nuclear reactor either. But we know how nuclear fission works.
ben_w · 1h ago
We have the Navier–Stokes equations which fit on a matchbox, yet for the last 25 years there's been a US$1,000,000 prize on offer to the first person providing a solution for a specific statement of the problem:
Prove or give a counter-example of the following statement:
In three space dimensions and time, given an initial velocity field, there exists a vector velocity and a scalar pressure field, which are both smooth and globally defined, that solve the Navier–Stokes equations.
ffwd · 3h ago
I think humans have some kind of algorithm for deciding what's true and consolidating information. What that is I don't know.
fellowniusmonk · 2h ago
This paper is about the limits in current systems.
Ai currently has issues with seeing what's missing. Seeing the negative space.
When dealing with complex codebases you are newly exposed to you tackle an issue from multiple angles. You look at things from data structures, code execution paths, basically humans clearly have some pressure to go, fuck, I think I lost the plot, and then approach it from another paradigm or try to narrow scope, or based on the increased information the ability to isolate the core place edits need to be made to achieve something.
Basically the ability to say, "this has stopped making sense" and stop or change approach.
Also, we clearly do path exploration and semantic compression in our sleep.
We also have the ability to transliterate data between semantic to visual structures, time series, light algorithms (but not exponential algorithms, we have a known blindspot there).
Humans are better at seeing what's missing, better at not closuring, better at reducing scope using many different approaches and because we operate in linear time and there are a lot of very different agents we collectively nibble away at complex problems over time.
I mean on a 1:1 teleomere basis, due to structural differences people can be as low as 93% similar genetically.
We also have different brain structures, I assume they don't all function on a single algorithmic substrate, visual reasoning about words, semantic reasoning about colors, synesthesia, the weird handoff between hemispheres, parts of our brain that handle logic better, parts of our brain that handle illogic better. We can introspect on our own semantic saturation, we can introspect that we've lost the plot. We get weird feelings when something seems missing logically, we can dive on that part and then zoom back out.
There's a whole bunch of shit the brain does because it has a plurality of structures to handle different types of data processing and even then the message type used seems flexible enough that you can shove word data into a visual processor part and see what falls out, and this happens without us thinking about it explicitly.
ffwd · 2h ago
Yep definitely agree with this.
ICBTheory · 3h ago
I guess so too... but whatever it is: it cannot possibly be something algorithmic. Therefore it doesn't matter in terms of demonstrating that AI has a boundary there, that cannot be transcended by tech, compute, training, data etc.
ffwd · 2h ago
Why can't it be algorithmic?
If the brain uses the same process on all information, then that is an algorithmic process. There is some evidence that it does do the same process to do things like consolidating information, processing the "world model" and so on.
Some processes are undoubtedly learned from experience but considering people seem to think many of the same things and are similar in many ways it remains to be seen whether the most important parts are learned rather than innate from birth.
xeonmc · 3h ago
I think the latter fact is quite self-demonstrably true.
mort96 · 3h ago
I would really like to see your definition of general intelligence and argument for why humans don't fit it.
ninetyninenine · 3h ago
Colloquially anything that matches humans in general intelligence and is built by us is by definition an agi and generally intelligent.
Humans are the bar for general intelligence.
umanwizard · 3h ago
How so?
deadbabe · 3h ago
First of all, math isn’t real any more than language isn’t real. It’s an entirely human construct, so it’s possible you cannot reach AGI using mathematical means, as math might not be able to fully express it. It’s similar to how language cannot fully describe what a color is, only vague approximations and measurements. If you wanted to create the color green, you cannot do it by describing various properties, you must create the actual green somehow.
hnfong · 3h ago
As a somewhat colorblind person, I can tell you that the "actual green" is pretty much a lie :)
It's a deeply philosophical question what constitutes a subjective experience of "green" or whatever... but intelligence is a bit more tractable IHO.
Workaccount2 · 3h ago
I don't think it would be unfair to accept the brain state of green as an accurate representation of green for all intents and purposes.
Similar to how "computer code" and "video game world" are the same thing. Everything in the video game world is perfectly encoded in the programming. There is nothing transcendent happening, it's two different views of the same core object.
like_any_other · 3h ago
Fair enough. But then, AGI wouldn't really be based on math, but on physics. Why would an artificially-constructed physical system have (fundamentally) different capabilities than a natural one?
ImHereToVote · 3h ago
Humans use soul juice to connect to the understandome. Machines can't connect to the understandome because of Gödels incompleteness, they can only make relationships between tokens. Not map them to reality like we can via magic.
My take is that it transcends any science that we'll understand and harness in the lifetime of anyone living today. It for all intents and purposes transcends science from our point of view, but not necessarily in principle.
lexicality · 3h ago
> are humans not generally intelligent?
Have you not met the average person on the street? (/s)
ben_w · 1h ago
Noted /s, but truly this is why I think even current models are already more disruptive than naysayers are willing to accept that any future model ever could be.
moktonar · 3h ago
Technically this is linked to the ability to simulate our universe efficiently. If it’s simulable efficiently then AGI is possible for sure, otherwise we don’t know. Everything boils down to the existence or not of an efficient algorithm to simulate Quantum Physics. At the moment we don’t know any except using QP itself (essentially hacking the Universe’s algorithm itself and cheating) with Quantum Computing (that IMO will prove exponentially difficult to harness, at least the same difficulty as creating AGI). So, yes, brains might be > computers.
agitracking · 3h ago
I always wondered how much of human intelligence can be mapped to mathematics.
Without reading the paper how the heck is agi mathematically impossible if humans are possible? Unless the paper is claiming humans are mathematically impossible?
I’ll read the paper but the title comes off as out of touch with reality.
geor9e · 3h ago
The title is clickbait. He more ends up saying that AGI is practically impossible today, given all our current paradigms of how we build computers, algorithms, and neural networks. There's an exponential explosion in how much computation time it requires to match the out-of-frame leaps and bounds that a human brain can make with just a few watts of power, and researchers have no clever ideas yet for emulating that trait.
fellowniusmonk · 2h ago
In the abstract it explicitly says current systems, the title is 100% click bait.
alganet · 3h ago
What makes you think that human intelligence is based on mathematics?
like_any_other · 3h ago
Because it's based on physics, which is based on mathematics. Alternately, even if we one day learn that physics is not reducible to mathematics, both humans and computers are still based on the same physics.
alganet · 3h ago
You're mistaking the thing for the tool we use to describe the thing.
Physics gives us a way to answer questions about nature, but it is not nature itself. It is also, so far (and probably forever), incomplete.
Math doesn't need to agree with nature, we can take it as far as we want, as long as it doesn't break its own rules. Physics uses it, but is not based on it.
sampl3username · 3h ago
And the soul?
mort96 · 3h ago
I will answer under the metaphysical assumption that there is no immaterial "soul", and that the entirety of the human experience arises from material things governed by the laws of physics. If you disagree with this assumption, there is no conversation to be had.
The laws of physics can, as far as I can tell, be described using mathematics. That doesn't mean that we have a perfect mathematical model of the laws of physics yet, but I see no reason to believe that such a mathematical model shouldn't be possible. Existing models are already extremely good, and the only parts which we don't yet have essentially perfect mathematical models for yet are in areas which we don't yet have the equipment necessary to measure how the universe behaves. At no point have we encountered a sign that the universe is governed by laws which can't be expressed mathematically.
This necessarily means that everything in the universe can also be described mathematically. Since the human experience is entirely made up of material stuff governed by these mathematical laws (as per the assumption in the first paragraph), human intelligence can be described mathematically.
Now there's one possible counter to this: even if we can perfectly describe the universe using mathematics, we can't perfectly simulate those laws. Real simulations have limitations on precision, while the universe doesn't seem to. You could argue that intelligence somehow requires the universe's seemingly infinite precision, and that no finite-precision simulation could possibly give rise to intelligence. I would find that extremely weird, but I can't rule it out a priori.
I'm not a physicist, and I don't study machine intelligence, nor organic intelligence, so I may be missing something here, but this is my current view.
DougN7 · 1h ago
I wonder if we could ever compute which exact atom in nuclear fission will split at a very specific time. If that is impossible, then our math and understanding of physics is so far short of what is needed that I don’t feel comfortable with your starting assumption.
mort96 · 39m ago
Quantum mechanics doesn't work like that. It doesn't describe when something will happen, but the evolution of branching paths and their probabilities.
alganet · 2h ago
I'm not talking about soul.
I'm just saying you're mistaking the thing for the the tool we use to describe the thing.
I'm also not talking about simulations.
Epistemologically, I'm talking about unknown unknowns. There are things we don't know, and we still don't know we don't know yet. Math and physics deal with known unknowns (we know we don't know) and known knowns (we know we know) only. Math and physics do not address unknown unknowns up until they become known unknowns (we did not tackle quantum up until we discover quantum).
We don't know how humans think. It is a known unknown, tackled by many sciences, but so far, incomplete in its description. We think we have a good description, but we don't know how good it is.
mort96 · 2h ago
If a human body is intelligent, and we could in principle set up a computer-simulated universe which has a human body in it and simulate it forward with sufficient accuracy to make the body operate as a real-world human body has, we would have an artificial general intelligence simulated by a computer (i.e using mathematics).
If you think there are potential flaws in this line of reasoning other than the ones I already covered, I'm interested to hear.
alganet · 2h ago
We currently can't simulate the universe. Not only in capability, but also knowledge. For example, we don't know where or when life started. Can't "simulate forward" from an event we don't understand.
Also, a simulation is not the thing. It's a simulation of the thing. See? The same issue. You're mistaking the thing for the tool we use to simulate the thing.
You could argue that the universe _is_ a simulation, or computational in nature. But that's speculation, not very different epistemologically from saying that a magic wizard made everything.
mort96 · 2h ago
Of course we can't simulate the universe (or, well, a slice of a universe which obeys the same laws as ours) right now, but we're discussing whether it's possible in principle or not.
I don't understand what fundamental difference you see between a thing governed by a set of mathematical laws and an implementation of a simulation which follows the same mathematical laws. Why would intelligence be possible in the former but fundamentally impossible in the latter, aside from precision limitations?
FWIW, nothing I've said assumes that the universe is a simulation, and I don't personally believe it is.
alganet · 12m ago
> a thing governed by a set of mathematical laws
Again, you're mistaking the thing for the tool we use to describe the thing.
> aside from precision limitations
It's not only about precision. There are things we don't know.
--
I think the universe always obeys rules for everything, but it's an educated guess. There could be rules we don't yet understand and are outside of what mathematics and physics can know. Again, there are many things we don't know. "We'll get there" is only good enough when we get there.
The difference is subtle. I require proof, you seem to be ok with not having it.
furyofantares · 3h ago
The first example of a problem that can't be solved by an algorithm is a wife asking her husband if she's gained weight.
I hate "stopped reading at x" type comments but, well, I did. For those who got further, is this paper interesting at all?
weregiraffe · 3h ago
Warning: this is quackery.
JdeBP · 3h ago
This has a single author; is not peer-reviewed; is not published in a journal; and was self-submitted both to PhilArchive and here on Hacker News.
pvg · 3h ago
There's nothing wrong with any of that, for an HN submission. The paper itself could be bad but that's what the discussion thread is for - discussing the thing presented rather than its meta attributes.
JdeBP · 2h ago
And no-one said that there was anything wrong, the inference being yours. But it's important to bear provenance in mind, and not get carried away by something like this more than one would be carried away by, say, an article on Medium propounding the same thing, as the bars to be cleared are about the same height.
pvg · 2h ago
The provenance is there for everyone to see so the purpose of the comment, beside some sort of implied aspersion is unclear.
JdeBP · 1h ago
The aspersions are yours and yours alone. And the provenance far from being apparent actually took some effort to discern, as it involves checking out whether and what sort of editorial board was involved for one thing, as well as looking for review processes and submission guidelines. You should ask yourself why you think so badly of Show HN posts, as you so clearly do, that when it's pointed out that such is the case you yourself directly leap to the idea that it's bad when no-one but you says any such thing.
ben_w · 1h ago
FWIW, I've never heard of PhilArchive before, so had no frame of reference for ease of self-publishing to it.
The idea is called IOpenER: Information Opens, Entropy Rises. It builds on Shannon’s information theory to show that in specific problem classes (those with α ≤ 1), adding information doesn’t reduce uncertainty — it increases it. The system can’t converge, because meaning itself keeps multiplying.
The core concept — entropy divergence in these spaces — was already present in my earlier paper, uploaded to PhilArchive on June 1. This version formalizes it. Apple’s study, The Illusion of Thinking, was published a few days later. It shows that frontier reasoning models like Claude 3.7 and DeepSeek-R1 break down exactly when problem complexity increases — despite adequate inference budget.
I didn’t write this paper in response to Apple’s work. But the alignment is striking. Their empirical findings seem to match what IOpenER predicts.
Curious what this community thinks: is this a meaningful convergence, or just an interesting coincidence?
Links:
This paper (entropy + IOpenER): https://philarchive.org/archive/SCHAIM-14
First paper (ICB + computability): https://philpapers.org/archive/SCHAII-17.pdf
Apple’s study: https://machinelearning.apple.com/research/illusion-of-think...
That said, the most obvious objection that comes to mind about the title is that … well, I feel that I’m generally intelligent, and therefore general intelligence of some sort is clearly not impossible.
Can you give a short précis as to how you are distinguishing humans and the “A” in artificial?
Well, given the specific way you asked that question I confirm your self assertion - and am quite certain that your level of Artificiality converges to zero, which would make you a GI without A...
- You stated to "feel" generally intelligent (A's don't feel and don't have an "I" that can feel) - Your nuanced, subtly ironic and self referential way of formulating clearly suggests that you are not a purely algorithmic entity
A "précis" as you wished: Artificial — in the sense used here (apart from the usual "planfully built/programmed system" etc.) — algorithmic, formal, symbol-bound.
Humans as "cognitive system" have some similar traits of course - but obviously, there seems to be more than that.
As you note in 2.1, there is widespread disagreement on what "AGI" means. I note that you list several definitions which are essentially "is human equivalent". As humans can be reduced to physics, and physics can be expressed as a computer program, obviously any such definition can be achieved by a sufficiently powerful computer.
For 3.1, you assert:
"""
Now, let's observe what happens when an Al system - equipped with state-of-the-art natural language processing, sentiment analysis, and social reasoning - attempts to navigate this question. The Al begins its analysis:
• Option 1: Truthful response based on biometric data → Calculates likely negative emotional impact → Adjusts for honesty parameter → But wait, what about relationship history? → Recalculating...
• Option 2: Diplomatic deflection → Analyzing 10,000 successful deflection patterns → But tone matters → Analyzing micro-expressions needed → But timing matters → But past conversations matter → Still calculating...
• Option 3: Affectionate redirect → Processing optimal sentiment → But what IS optimal here? The goal keeps shifting → Is it honesty? Harmony? Trust? → Parameters unstable → Still calculating...
• Option n: ....
Strange, isn't it? The Al hasn't crashed. It's still running. In fact, it's generating more and more nuanced analyses. Each additional factor may open ten new considerations. It's not getting closer to an answer - it's diverging.
"""
Which AI? ChatGPT just gives an answer. Your other supposed examples have similar issues in that it looks like you've *imagined* an AI rather than having tried asking an AI to seeing what it actually does or doesn't do.
I'm not reading 47 pages to check for other similar issues.
No it doesn’t.
Shannon entropy measures statistical uncertainty in data. It says nothing about whether an agent can invent new conceptual frames. Equating “frame changes” with rising entropy is a metaphor, not a theorem, so it doesn’t even make sense as a mathematical proof.
This is philosophical musing at best.
But the paper doesn’t just restate Shannon.
It extends this very formalism to semantic spaces where the symbol set itself becomes unstable. These situations arise when (a) entropy is calculated across interpretive layers (as in LLMs), and (b) the probability distribution follows a heavy-tailed regime (α ≤ 1). Under these conditions, entropy divergence becomes mathematically provable.
This is far from being metaphorical: it’s backed by formal Coq-style proofs (see Appendix C in he paper).
AND: it is exactly the mechanism that can explain the Apple-Papers' results
Still an interesting take and will need to dive in more, but already if we assume the brain is doing information processing then the immediate question is how can the brain avoid this problem, as others are pointing out. Is biological computation/intelligence special?
Humans don't have the processing power to traverse such vast spaces. We use heuristics, in the same way a chess player does not iterate over all possible moves.
It's a valid point to make, however I'd say this just points to any AGI-like system having the same epistemological issues as humans, and there's no way around it because of the nature of information.
Stephen Wolfram's computational irreducibility is another one of the issues any self-guided, phyiscally grounded computing engine must have. There are problems that need to be calculated whole. Thinking long and hard about possible end-states won't help. So one would rather have 10000 AGIs doing somewhat similar random search in the hopes that one finds something useful.
I guess this is what we do in global-scale scientific research.
AI via LLMs has limitations, but they don't come from computability.
[1] https://sortingsearching.com/2021/07/18/roger-penrose-ai-ske...
But this isn’t that, as I’m not making a claim about consciousness or invoking quantum physics or microtubules (which, I agree, are highly speculative).
The core of my argument is based on computability and information theory — not biology. Specifically: that algorithmic systems hit hard formal limits in decision contexts with irreducible complexity or semantic divergence, and those limits are provable using existing mathematical tools (Shannon, Rice, etc.).
So in some way, this is the non-microtubule version of AI critique. I don’t have the physics background to engage in Nobel-level quantum speculation — and, luckily, it’s not needed here.
> Anyway, this is not part of the questions this paper seeks to answer. Neither will we wonder in what way it could make sense to measure the strength of a model by its ability to find its relative position to the object it models. Instead, we chose to stay ignorant - or agnostic? - and take this fallible system called "human". As a point of reference.
Cowards.
That's the main counter argument and acknowledging its existence without addressing it is a craven dodge.
Assuming the assumptions[1] are true, then human intelligence isn't even able to be formalized under the same pretext.
Either human intelligence isn't
1. Algorithmic. The main point of contention. If humans aren't algorithmically reducible - even at the level computation of physics, then human cognition is supernatural.
2. Autonomous. Trivially true given that humans are the baseline.
3. Comprehensive (general): Trivially true since humans are the baseline.
4. Competent: Trivially true given humans are the baseline.
I'm not sure how they reconcile this given that they simply dodge the consequences that it implies.
Overall, not a great paper. It's much more likely that their formalism is wrong than their conclusion.
Footnotes
1. not even the consequences, unfortunately for the authors.
–Are we treating an arbitrary ontological assertion as if it’s a formal argument that needs to be heroically refuted? Or better: is that metaphysical setup an argument?
If that’s the game, fine. Here we go:
– The claim that one can build a true, perfectly detailed, exact map of reality is… well... ambitious. It sits remarkably far from anything resembling science , since it’s conveniently untouched by that nitpicky empirical thing called evidence. But sure: freed from falsifiability, it can dream big and give birth to its omnicartographic offspring.
– oh, quick follow-up: does that “perfect map” include itself? If so... say hi to Alan Turing. If not... well, greetings to Herr Goedel.
– Also: if the world only shows itself through perception and cognition, how exactly do you map it “as it truly is”? What are you comparing your map to — other observations? Another map?
– How many properties, relations, transformations, and dimensions does the world have? Over time? Across domains? Under multiple perspectives? Go ahead, I’ll wait... (oh, and: hi too.. you know who)
And btw the true detailed map of the world exists.... It’s the world.
It’s just sort of hard to get a copy of it. Not enough material available ... and/or not enough compute....
P.S. Sorry if that came off sharp — bit of a spur-of-the-moment reply. If you want to actually dig into this seriously, I’d be happy to.
More interestingly, humans are capable of assessing the results of their "neural misfires" ("hmm, there's something to this"), whereas even if we could make a computer do such mistakes, it wouldn't know its Penny Lane from its Daddy's Car[0], even if it managed to come up with one.
[0]https://www.youtube.com/watch?v=LSHZ_b05W7o
And we can get LLMs to do better by just prompting them to "think step by step" or replacing the first ten attempts to output a "stop" symbolic token with the token for "Wait… "?
No comments yet
Well, it in fact depends on what intelligence is to your understanding:
-If it intelligence = IQ, i.e. the rational ability to infer, to detect/recognize and extrapolate patterns etc, then AI is or will soon be more intelligent than us, while we humans are just muddling through or simply lucky having found relativity theory and other innovations just at the convenient moment in time ... So then, AI will soon also stumble over all kind of innovations. None of both will be able to deliberately think beyond what is thinkable at the respective present.
- But If intelligence is not only a level of pure rational cognition, but maybe an ability to somehow overcome these frame-limits, then humans obviously exert some sort of abilities that are beyond rational inference. Abilities that algorithms can impossibly reach, as all they can is compute.
- Or: intelligence = IQ, but it turns out to be useless in big, pivotal situations where you’re supposed to choose the “best” option — yet the set of possible options isn’t finite, knowable, or probabilistically definable. There’s no way to defer to probability, to optimize, or even to define what “best” means in a stable way. The whole logic of decision collapses — and IQ has nothing left to grab onto.
The main point is: neither algorithms nor rationality can point beyond itself.
In other words: You cannot think out of the box - thinking IS the box.
(maybe have a quick look at my first proof -last chapter before conclusion- - you will find a historical timeline on that IQ-Thing)
All of that just sounds hard, not mathematically impossible.
As I understand it, this is mostly a rehash on the dated Lucas Penrose argument, which most Mind Theory researches refute.
You make broadly valid points, particularly about the advantages of embodyment, but I just dont think theyre good responses to the theoretical article under discussion (or the comment that you were responding to).
So because of this we know reality is governed by maths. We just can’t fully model the high level consequence of emergent patterns due to the sheer complexity of trillions of interacting atoms.
So it’s not that there’s some mysterious supernatural thing we don’t understand. It’s purely a complexity problem in that we only don’t understand it because it’s too complex.
What does humility have to do with anything?
> So because of this we know reality is governed by maths.
That's not really true. You have a theory, and let's presume so far it's consistent with observations. But it doesn't mean it's 100% correct, and doesn't mean at some point in the future you won't observe something that invalidates the theory. In short, you don't know whether the theory is absolutely true and you can never know.
Without an absolutely true theory, all you have is belief or speculation that reality is governed by maths.
> What does humility have to do with anything?
Not the GP but I think humility is kinda relevant here.
Let me repharse it. As far as we know all of reality is governed by the principles of logic and therefore math. This is the most likely possibility and we have based all of our technology and culture and science around this. It is the fundamental assumption humanity has made on reality. We cannot consistently demonstrate disproof against this assumption.
>Not the GP but I think humility is kinda relevant here.
How so? If I assume all of reality is governed by math, but you don't. How does that make me not humble but you humble? Seems personal.
Speak for yourself. LLMs are a feedforward algorithm inferring static weights to create a tokenized response string.
We can compare that pretty trivially to the dynamic relationship of neurons and synapses in the human brain. It's not similar, case closed. That's the extent of serious discussion that can be had comparing LLMs to human thought, with apologies to Chomsky et. al. It's like trying to find the anatomical differences between a medieval scribe and a fax machine.
If we're OK with descriptions so lossy that they fit in a sentence, we also understand the human brain:
A electrochemical network with external inputs and some feedback loops, pumping ions around to trigger voltage cascades to create muscle contractions as outputs.
No. The answer is already solved; AI is not a brain, we can prove this by characteristically defining them both and using heuristic reasoning.
That "can" should be "could", else it presumes too much.
For both human brains and surprisingly small ANNs, far smaller than LLMs, humanity collectively does not yet know the defining characteristics of the aspects we care about.
I mean, humanity don't agree with itself what any of the three initials of AGI mean, there's 40 definitions of the word "consciousness", there are arguments about if there is either exactly one or many independent G-factors in human IQ scores, and also if those scores mean anything beyond correlating with school grades, and human nerodivergence covers various real states of existance that many of us find incomprehensible (sonetimes mutually, see e.g. most discussions where aphantasia comes up).
The main reason I expect little from an AI is that we don't know what we're doing. The main reason I can't just assume the least is because neither did evolution when we popped out.
https://www.reddit.com/r/singularity/comments/1lbbg0x/geoffr...
https://youtu.be/qrvK_KuIeJk?t=284
In that video above George Hinton, directly says we don't understand how it works.
So I don't speak just for myself. I speak for the person who ushered in the AI revolution, I speak for Experts in the field who know what they're talking aboutt. I don't speak for people who don't know what they're talking about.
Even though we know it's a feedforward network and we know how the queries are tokenized you cannot tell me what an LLM would say nor tell me why an LLM said something for a given prompt showing that we can't fully control an LLM because we don't fully understand it.
Don't try to just argue with me. Argue with the experts. Argue with the people who know more than you, Hinton.
And that's okay - his humility isn't holding anyone back here. I'm not claiming to have memorized every model weight ever published, either. But saying that we don't know how AI works is empirically false; AI genuinely wouldn't exist if we weren't able to interpret and improve upon the transformer architecture. Your statement here is a dangerous extrapolation.
> you cannot tell me what an LLM would say nor tell me why an LLM said something for a given prompt showing that we can't fully control an LLM because we don't fully understand it.
You'd think this, but it's actually wrong. If you remove all of the seeded RNG during inference (meaning; no random seeds, no temps, just weights/tokenizer), you can actually create an equation that deterministically gives you the same string of text every time. It's a lot of math, but it's wholly possible to compute exactly what AI would say ahead of time if you can solve for the non-deterministic seeded entropy, or remove it entirely.
LLM weights and tokenizer are both always idempotent, the inference software often introduces variability for more varied responses. Just so we're on the same page here.
Care to elaborate? Because that is utter nonsense.
"Cat" lights up a certain set of neurons, but then "cat" looks completely different. That is what we don't really understand.
(This is an illustrative example made for easy understanding, not something I specifically went and compared)
We don't and can't know with certainty which specific atoms will fission in a nuclear reactor either. But we know how nuclear fission works.
Ai currently has issues with seeing what's missing. Seeing the negative space.
When dealing with complex codebases you are newly exposed to you tackle an issue from multiple angles. You look at things from data structures, code execution paths, basically humans clearly have some pressure to go, fuck, I think I lost the plot, and then approach it from another paradigm or try to narrow scope, or based on the increased information the ability to isolate the core place edits need to be made to achieve something.
Basically the ability to say, "this has stopped making sense" and stop or change approach.
Also, we clearly do path exploration and semantic compression in our sleep.
We also have the ability to transliterate data between semantic to visual structures, time series, light algorithms (but not exponential algorithms, we have a known blindspot there).
Humans are better at seeing what's missing, better at not closuring, better at reducing scope using many different approaches and because we operate in linear time and there are a lot of very different agents we collectively nibble away at complex problems over time.
I mean on a 1:1 teleomere basis, due to structural differences people can be as low as 93% similar genetically.
We also have different brain structures, I assume they don't all function on a single algorithmic substrate, visual reasoning about words, semantic reasoning about colors, synesthesia, the weird handoff between hemispheres, parts of our brain that handle logic better, parts of our brain that handle illogic better. We can introspect on our own semantic saturation, we can introspect that we've lost the plot. We get weird feelings when something seems missing logically, we can dive on that part and then zoom back out.
There's a whole bunch of shit the brain does because it has a plurality of structures to handle different types of data processing and even then the message type used seems flexible enough that you can shove word data into a visual processor part and see what falls out, and this happens without us thinking about it explicitly.
Some processes are undoubtedly learned from experience but considering people seem to think many of the same things and are similar in many ways it remains to be seen whether the most important parts are learned rather than innate from birth.
Humans are the bar for general intelligence.
It's a deeply philosophical question what constitutes a subjective experience of "green" or whatever... but intelligence is a bit more tractable IHO.
Similar to how "computer code" and "video game world" are the same thing. Everything in the video game world is perfectly encoded in the programming. There is nothing transcendent happening, it's two different views of the same core object.
https://ai.vixra.org/pdf/2506.0065v1.pdf
Have you not met the average person on the street? (/s)
Also, interesting timing of this post - https://news.ycombinator.com/item?id=44348485
I’ll read the paper but the title comes off as out of touch with reality.
Physics gives us a way to answer questions about nature, but it is not nature itself. It is also, so far (and probably forever), incomplete.
Math doesn't need to agree with nature, we can take it as far as we want, as long as it doesn't break its own rules. Physics uses it, but is not based on it.
The laws of physics can, as far as I can tell, be described using mathematics. That doesn't mean that we have a perfect mathematical model of the laws of physics yet, but I see no reason to believe that such a mathematical model shouldn't be possible. Existing models are already extremely good, and the only parts which we don't yet have essentially perfect mathematical models for yet are in areas which we don't yet have the equipment necessary to measure how the universe behaves. At no point have we encountered a sign that the universe is governed by laws which can't be expressed mathematically.
This necessarily means that everything in the universe can also be described mathematically. Since the human experience is entirely made up of material stuff governed by these mathematical laws (as per the assumption in the first paragraph), human intelligence can be described mathematically.
Now there's one possible counter to this: even if we can perfectly describe the universe using mathematics, we can't perfectly simulate those laws. Real simulations have limitations on precision, while the universe doesn't seem to. You could argue that intelligence somehow requires the universe's seemingly infinite precision, and that no finite-precision simulation could possibly give rise to intelligence. I would find that extremely weird, but I can't rule it out a priori.
I'm not a physicist, and I don't study machine intelligence, nor organic intelligence, so I may be missing something here, but this is my current view.
I'm just saying you're mistaking the thing for the the tool we use to describe the thing.
I'm also not talking about simulations.
Epistemologically, I'm talking about unknown unknowns. There are things we don't know, and we still don't know we don't know yet. Math and physics deal with known unknowns (we know we don't know) and known knowns (we know we know) only. Math and physics do not address unknown unknowns up until they become known unknowns (we did not tackle quantum up until we discover quantum).
We don't know how humans think. It is a known unknown, tackled by many sciences, but so far, incomplete in its description. We think we have a good description, but we don't know how good it is.
If you think there are potential flaws in this line of reasoning other than the ones I already covered, I'm interested to hear.
Also, a simulation is not the thing. It's a simulation of the thing. See? The same issue. You're mistaking the thing for the tool we use to simulate the thing.
You could argue that the universe _is_ a simulation, or computational in nature. But that's speculation, not very different epistemologically from saying that a magic wizard made everything.
I don't understand what fundamental difference you see between a thing governed by a set of mathematical laws and an implementation of a simulation which follows the same mathematical laws. Why would intelligence be possible in the former but fundamentally impossible in the latter, aside from precision limitations?
FWIW, nothing I've said assumes that the universe is a simulation, and I don't personally believe it is.
Again, you're mistaking the thing for the tool we use to describe the thing.
> aside from precision limitations
It's not only about precision. There are things we don't know.
--
I think the universe always obeys rules for everything, but it's an educated guess. There could be rules we don't yet understand and are outside of what mathematics and physics can know. Again, there are many things we don't know. "We'll get there" is only good enough when we get there.
The difference is subtle. I require proof, you seem to be ok with not having it.
I hate "stopped reading at x" type comments but, well, I did. For those who got further, is this paper interesting at all?