The Darwin Gödel Machine: AI that improves itself by rewriting its own code

I agree: if they could, they would be doing it already.

Case in point: one of the first things done once ChatGPT started getting popular was "auto-gpt"; roughly, let it loose and see what happens.

The same thing will happen to any accessible model in the future. Someone, somewhere will ask it to self-improve/make as much money as possible, with as little leashes as possible. Maybe even the labs themselves do that, as part of their post-training ops for new models.

Therefore, we can assume that if the existing models _could_ be doing that, they _would_ be doing that.

That doesn't say anything about new models released 6 months or 2 years from now.

I do wonder though if big labs are running this with model training episodes as well.

Let's ground that a bit.

Have a look at ARC AGI 1 challenge/benchmark. Solve a problem or two yourself. Know that ARC AGI 1 is practically solved by a few LLMs as of Q1 2025.

Then have a look at the ARC AGI 2 challenge. Solve a problem or two yourself. Note that as of today, it is unsolved by LLMs.

Then observe that the "difficulty" of ARC AGI 1 and 2 for a human are relatively the same but challenge 2 is much harder for LLMs than 1.

ARC AGI 2 is going to be solved *within* 12 months (my bet is on 6 months). If it's not, I'll never post about AI on HN again.

There's only one problem to solve, i.e. "how to make LLMs truly see like humans do". Right now, any vision based features that the models exhibit comes from maximizing the use of engineering (i.e. applying CNNs on image slices, chunks, maybe zooming and applying ocr, vector search etc), it isn't vision like ours and isn't a native feature for these models.

Once that's solved, then LLMs or new Algo will be able to use a computer perfectly by feeding it screen capture. End of white collar jobs 2-5 years after (as we know it).

Edit - added "(as we know it)". And fixed missing word.

realistically the best you could do is evolve the prompt. maybe you could change input data preprocessing?

anyways the idea of current llm architectures self-improving via its own code seems silly as there are surprisingly few knobs to turn, and it's ~super expensive to train.

as a side note it's impressive how resistant the current architecture is to incremental RL away from results, since if even one "undesired input" result is multiple tokens, the coupling between the tokens is difficult to disentangle. (how do you separate jinping from jin-gitaxias for example)

I'm not sure how much an agent could do though given the right tools. access to a task mgt system, test tracker. robust requirements/use cases.

Who is claiming anything can self improve exponentially?

We will be on the way to AGI when your model can learn Python just by reading the Python docs...Once...

Even if they had human level intuition, they wouldn't be able to improve exponentially without human money, and they would need an exponentially growing amount of it to do so.

It’s not far off from human improvement. Our improvement is limited to what we can remember as well.

We go a bit further in the sense that the neural network itself can grow new modules.

That being said they can write thousands of lines an hour and can probably do things that would be impossible for a human. (Imagine having the LLM skip code and spit out compiled binaries as one example)

I think this happens with humans in places like social media echo chambers (or parts of academia) when they talk and talk and talk a whole lot without contact with any outer reality. It can be a source of creativity but also madness and insane ideas.

I’m quite firmly on the side of learning requiring either direct or indirect (informed by others) embodiment, or at least access to something outside. I don’t think a closed system can learn, and I suspect that this may reflect the fact that entropy increases in a closed system (second law).

As I said recently in another thread, I think self contemplating self improving “foom” AI scenarios are proposing informatic perpetual motion or infinite energy machines.

Everything has to “touch grass.”

  Problem: 
    1) we want to train on GitHub repos
    2) most datasets are spoiled. Training on GitHub would definitely spoil

  Solution:
    Hand write new problems!!!
    ... leetcode style ....
    ... and we'll check if it passes test

  Example:
    What's the decimal part of this float?

Sure in all of GitHub we can filter such code out by ngram!

Maybe my favorite part is that it has 60 authors and became the de facto benchmark for awhile

The expressivity is there, the only question is how to encode useful functions into those weights, especially when we don’t know how to write those functions by hand.

[0] http://neuralnetworksanddeeplearning.com/chap1.html

Or yeah if it can modify its own weights sensibly, which feels ... impossible really.

What's the difference?

Give it some serious thought. Challenge whichever answer you come up with. I guarantee this will be trickier than you think

Then if you go further and alter the architecture by introducing clean category theory morphisms and build from there you can have a dynamic network - but you will still have to retrain this network every time you change the structure.

You can spin this further and know the need for a real-world training set and a loss function that will have to competete against other networks. In the end a human brain is already best at this and embodied in the real world.

What i want to add here is that our neurons not take in weights - they also fire depending on whether one input comes after another or before and differs down to the nanoseconds here - unmatched in IT and ofc heaps more efficient.

I still would say its possible though and currently work on 4D lifeforms built on dynamic compute graphs that can do this in a set virtual environment.

So this is pretty awesome stuff, but its a long fetch from anything we do right now.

There are two separate aspects here. In this paper they improve the software around the model, not the model itself. What they're saying is that the software improvements carried over to other models, so it wasn't just optimising around model-specific quirks.

What you're describing with training large LLMs first is usually called "distillation" and it works on training the smaller LLM to match the entire distribution of tokens at once (hence it's faster in practice).

For this part of the stack the interesting question to me is how to identify and mitigate.

We need more showmanship, more dramatic catastrophizing. I feel like our current crop of doomers isn’t quite shameless enough to be really entertaining.

https://www.alignmentforum.org/posts/6Xgy6CAf2jqHhynHL/what-...

I don't think scaling this to also run training runs with the models is something that small labs / phd students can do. They lack the compute for that by orders of magnitude. Trying it with toy models might not work, trying it with reasonably large models is out of their budget. The only ones who can realistically do this are large labs (goog, oai, meta, etc.)