AGI is an engineering problem, not a model training problem

So I don't buy the engineering angle, I also don't think LLMs will scale up to AGI as imagined by Asimov or any of the usual sci-fi tropes. There is something more fundamental missing, as in missing science, not missing engineering.

The architecture has to allow for gradient descent to be a viable training strategy, this means no branching (routing is bolted on).

And the training data has to exist, you can't find millions of pages depicting every thought a person went through before writing something. And such data can't exist because most thoughts aren't even language.

Reinforcement learning may seem like the answer here: bruteforce thinking to happen. But it's grossly sample-inefficient with gradient descent and therefore only used for finetuning.

LLM's are regressive models and the configuration that was chosen where every token can only look back allows for very sample-efficient training (one sentence can be dozens of samples).

Right now, LLMs feel like they’re at the same stage as raw FLOPs; impressive, but unwieldy. You can already see the beginnings of "systems thinking" in products like Claude Code, tool-augmented agents, and memory-augmented frameworks. They’re crude, but they point toward a future where orchestration matters as much as parameter count.

I don’t think the "bitter lesson" and the "engineering problem" thesis are mutually exclusive. The bitter lesson tells us that compute + general methods win out over handcrafted rules. The engineering thesis is about how to wrap those general methods in scaffolding that gives them persistence, reliability, and composability. Without that scaffolding, we’ll keep getting flashy demos that break when you push them past a few turns of reasoning.

So maybe the real path forward is not "bigger vs. smarter," but bigger + engineered smarter. Scaling gives you raw capability; engineering decides whether that capability can be used in a way that looks like general intelligence instead of memoryless autocomplete.

Just hand waving some “distributed architecture” and trying to duct tape modules together won’t get us any closer to AGI.

The building blocks themselves, the foundation, has to be much better.

Arguably the only building block that LLMs have contributed is that we have better user intent understanding now; a computer can just read text and extract intent from it much better than before. But besides that, the reasoning/search/“memory” are the same building blocks of old, they look very similar to techniques of the past, and that’s because they’re limited by information theory / computer science, not by today’s hardware or systems.

What is that? What could merely require light elementary education and then it takes off and self improves to match and surpass us? That would be artificial comprehension, something we've not even scratched. AI and trained algorithms are "universal solvers" given enough data, This AGI would be something different, this is understanding, comprehending. Instantaneous decomposition of observations for assessment of plausibility, and then recombination for assessment of combination plausibility - all continual and instant for assessment of personal safety: all that happens in people continually while awake. Be that monitoring of personal safety be for physical or loss of client during sales negotiation. Our comprehending skills are both physical and abstract. This requires a dynamic assessment, an ongoing comprehension that is validating observations as a foundation floor, so a more forward train of thought, a "conscious mind" can make decisions without conscious thought about lower level issues like situational safety. AGI needs all that dynamic comprehending capability, to satisfy its name of being general.

A good contrast is quantum computing. We know that's possible, even feasible, and now are trying to overcome the engineering hurdles. And people still think that's vaporware.

It can plan and take actions towards arbitrary goals in a wide variety of mostly text-based domains. It can maintain basic "memory" in text files. It's not smart enough to work on a long time horizon yet, it's not embodied, and it has big gaps in understanding.

But this is basically what I would have expected v1 to look like.

Brains are continuous - they don’t stop after processing one set of inputs, until a new set of inputs arrives.

Brains continuously feed back on themselves. In essence they never leave training mode although physical changes like myelination optimize the brain for different stages of life.

Brains have been trained by millions of generations of evolution, and we accelerate additional training during early life. LLMs are trained on much larger corpuses of information and then expected to stay static for the rest of their operational life; modulo fine tuning.

Brains continuously manage context; most available input is filtered heavily by specific networks designed for preprocessing.

I think that there is some merit that part of achieving AGI might involve a systems approach, but I think AGI will likely involve an architectural change to how models work.

But I still see all the same debates around AGI - how do we define it? what components would it require? could we get there by scaling or do we have to do more? and so on.

I don't see anyone addressing the most truly fundamental question: Why would we want AGI? What need can it fulfill that humans, as generally intelligent creatures, do not already fulfill? And is that moral, or not? Is creating something like this moral?

We are so far down the "asking if we could but not if we should" railroad that it's dazzling to me, and I think we ought to pull back.

In the meantime I guess all the AI companies will just keep burning compute to get marginal improvements. Sounds like a solid plan! The craziest thing about all of this is that ML researchers should know better!! Anyone with extensive experience training models small or large knows that additional training data offers asymptotic improvements.

Here, AGI is being described as an engineering problem, in contrast to a “model training” problem. That is, I think at least, he’s at least saying that more work needs to be done at an R&D level. I agree with those who are saying it is maybe not even an engineering problem yet, but should be noted that he’s at least pushing away from just running the existing programs harder, which seems to be the plan with trillions of dollars behind it.

I doubt very much we will ever build a machine that has perfect knowledge of the future or that can solve each and every “hard” reasoning problem, or that can complete each narrow task in a way we humans like. In other words, it’s not simply a matter of beating benchmarks.

In my mind at least, AGI’s definition is simple: anything that can replace any human employee. That construct is not merely a knowledge and reasoning machine, but also something that has a stake on its own work and that can be inserted in a shared responsibility graph. It has to be able to tell that senior dev “I know planning all the tasks one year in advance is busy-work you don’t want to do, but if you don’t, management will terminate me. So, you better do it, or I’ll hack your email and show everybody your porn subscriptions.”

Here are the metrics by which the author defines this plateau: "limited by their inability to maintain coherent context across sessions, their lack of persistent memory, and their stochastic nature that makes them unreliable for complex multi-step reasoning."

If you try to benchmark any proxy of the points above, for instance "can models solve problems that require multi steps in agentic mode" (PlanBench, BrowseComp, I've even built custom benchmarks), the progress between models is very clear, and shows no sign of slowing down.

And this does convert to real-world tasks : yesterday, I had GPT-5 build me complex react charts in one-shot, whereas previous models needed more constant supervision.

I think we're moving goalposts too fast for LLMs, that's what can lead us to believe they've plateaued : but just try using past models for your current tasks (you can use use open models to be sure they were not updated) and see them struggle.

All of our current approaches "emulate" but do not "execute" general intelligence. The damning paper above basically concludes they're incredible pattern matching machines, but thats about it.

Why some people understood when they tried it with blockchain, nfts, web3, AR, ... any good engineer should know principle of energy efficiency instead of having faith in the Infinite monkey theorem

We don’t even know how.

IME it’s both though. Better models, bigger models, and infrastructure all help get to AGI.

I see. So the author rejects the hypothesis of emergent behavior in LLM, but somehow thinks it will magically appear if the "engineering" is correct.

Self contradictory.

Trying to model AGI off how humans think, without including emotion as a fundamental building block, is like trying to build a computer that'll run without electricity. People are emotional beings first. So much of how we learn that something is good or bad is due to emotion.

In an AGI context that means:

  Happiness: how do I build an unguided feedback mechanism for reward?
  Fear: how do I build an unguided feedback mechanism to instruct to flee?
  Sadness: how do I build an unguided feedback mechanism to instruct to seek external support?
  Anger: how do I build an unguided feedback mechanism to push back on external entities that violate expectations?
  Disgust: how do I build an unguided feedback mechanism to instruct to avoid?

What if intelligence requires agency ?

Because if they don't, I honestly don't think they can approach AGI.

I have the feeling it's a common case of lack of humility from an entire field of science who refuses to look at other fields to understand what they're doing.

Not to mention how to define intelligence in evolution, epistemology, ontology, etc.

Approaching AI with a silicon valley mindset is not a good idea.

AGI would take making at least one full brain, and then putting many of those working together, efficiently.

I don't believe we can engineer our way out of that before explaining how the f. the wetware works first.

the idea that you would somehow produce intelligence by feeding billions of reddit comments into a statistical text model is will go down as the biggest con in history

(so far)

If we want to learn, look to nature, and it *has to be alive*.

AGI is poorly defined and thus is a science "problem", and a very low priority one at that.

No amount of engineering or model training is going to get us AGI until someone defines what properties are required and then researches what can be done to achieve them within our existing theories of computation which all computers being manufactured today are built upon.

Will AGI require ‘consciousness’, another poorly understood concept? How are mammalian brain even wired up? The most advanced model is the Allen Institute’s Mesoscale Connectivity Atlas which is at best a low resolution static roadmap, not a dynamic description of how a brain operates in real time. And it describes a mouse brain, not a human brain which is far, far more complex, both in terms of number of parts, and architecture.

People are just finally starting to acknowledge LLMs are dead ends. The effort expended on them over the last five years could well prove a costly diversion along the road to AGI, which is likely still decades in the future.

Intelligence must be built from a first principles theory of what intelligence actually is.

The missing science to engineer intelligence is composable program synthesis. Aloe (https://aloe.inc) recently released a GAIA score demonstrating how CPS dramatically outperforms other generalist agents (OpenAI's deep research, Manus, and Genspark) on tasks similar to those a knowledge worker would perform.

Continuing to want to make a non-deterministic system behave like a deterministic system will be interesting to watch.