It's always a treat to watch a Carmack lecture or read anything he writes, and his notes here are no exception. He writes as an engineer, for engineers and documents all his thought processes and misteps in the exact detailed yet concise way you'd want a colleague to who was handing off some work.
One question I would have about the research direction is the emphasis on realtime. If I understand correctly he's doing online learning in realtime. Obviously makes for a cool demo and pulls on his optimisation background, and no doubt some great innovations will be required to make this work. But I guess the bitter lesson and recent history also tell us that some solutions may only emerge at compute levels beyond what is currently possible for realtime inference let alone learning. And the only example we have of entities solving Atari games is the human brain, of which we don't have a clear understanding of the compute capacity. In which case, why wouldn't it be better to focus purely on learning efficiency and relax the realtime requirement for now?
That's a genuine question by the way, definitely not an expert here and I'm sure there's a bunch of value to working within these constraints. I mean, jumping spiders solve reasonably complex problems with 100k neurons, so who knows.
kilpikaarna · 3h ago
I'm sure there were offline rendering and 3D graphics workstation people saying the same about the comparatively crude work he was doing in the early 90s...
Obviously both Carmack and the rest of the world has changed since then, but it seems to me his main strength has always been in doing more with less (early id/Oculus, AA). When he's working in bigger orgs and/or with more established tech his output seems to suffer, at least in my view (possibly in his as well since he quit both Bethesda-id and Meta).
I don't know Carmack and can't claim to be anywhere close to his level, but as someone also mainly interested in realtime stuff I can imagine he also feels a slight disdain for the throw-more-compute-at-it approach of the current AI boom. I'm certainly glad he's not running around asking for investor money to train an LLM.
Best case scenario he teams up with some people who complement his skillset (akin to the game designers and artists at id back in the day) and comes up with a way to help bring some of the cutting edge to the masses, like with 3D graphics.
LarsDu88 · 1h ago
The thing about Carmack in the 90s...
There was a lot of research going on around 3d graphics. Companies like SGI and Pixar were building specialized workstations for doing vector operations for 3d rendering. 3d was a thing. Game consoles with specialized 3d hardware would launch in 1994 with the Sega Saturn and the Sony Playstation (in Japan only for one year)
What Carmack did was basically get a 3d game running on existing COMMODITY hardware. The 386 chip that most people used for their excel spreadsheets did not do floating point operations well, so Carmack figured out how to do everything using integers.
May 1992 -> Wolfenstein 3d releases
December 1993 -> Doom releases
December 1994 -> Sony Playstation launches in Japan
June 1996 -> Quake releases
So Wolfenstein and Doom were actually not really 3d games, but rather 2.5 games (you can't have rooms below other rooms). The first 3d game here is actually Quake which also eventually also got hardware acceleration support.
Carmack was the master of doing the seeminly impossible on super constrained hardware on virtually impossible timelines. If DOOM released in 1994 or 1995, would we still remember it in the same way?
hx8 · 33m ago
> If DOOM released in 1994 or 1995, would we still remember it in the same way?
Maybe. One aspect of Wolfenstein and Doom's popularity is that it was years ahead of everyone else technically on PC hardware. The other aspect is that they were genre defining titles that set the standards for gameplay design. I think Doom Deathmatch would have caught on in 1995, as there really were very few (just Command and Conquer?) standout PC network multiplayer games released between 1993 and 1995.
LarsDu88 · 27m ago
I guess the thing about rapid change is... it's hard to imagine what kind of games would exist in a DOOMless world in an alternate 1995.
The first 3d console games started to come out that year, like Rayman.
Star Wars Dark Forces with its own custom 3d engine also came out. Of course Dark Forces was, however, an overt clone of DOOM.
It's a bit ironic, but I think the gameplay innovation of DOOM tends to hold up more than the actual technical innovation. Things like BSP for level partitioning have slowly been phased out of game engines, we have ample floating point compute power and hardware acceleration ow, but even developers of the more recent DOOM games have started to realize that they should return to the original formula of "blast zombies in the face at high speed, and keep plot as window dressing"
CamperBob2 · 9m ago
If DOOM released in 1994 or 1995, would we still remember it in the same way?
I think so, because the thing about DOOM is, it was an insanely good game. Yes, it pioneered fullscreen real-time perspective rendering on commodity hardware, instantly realigning the direction of much of the game industry, yadda yadda yadda, but at the end of the day it was a good-enough game for people to remember and respect even without considering the tech.
Minecraft would be a similar example. Minecraft looked like total ass, and games with similar rendering technology could have been (and were) made years earlier, but Minecraft was also good. And that was enough.
johnb231 · 9h ago
From the notes:
"A reality check for people that think full embodied AGI is right around the corner is to ask your dancing humanoid robot to pick up a joystick and learn how to play an obscure video game."
ferguess_k · 5h ago
We don't really need AGI. We need better specialized AIs. Throw in a few specialized AIs and they will leave some impact in the society. That might not be that far away.
babyent · 2h ago
Why not just hire like 100 of the smartest people across domains and give them SOTA AI, to keep the AI as accurate as possible?
Each of those 100 can hire teams or colleagues to make their domain better, so there’s always human expertise keeping the model updated.
trial3 · 2h ago
"just"
babyent · 1h ago
They’re spending 10s of billions. Yes, just.
200 million to have dedicated top experts on hand is reasonable.
bluGill · 5h ago
Specialized AIs have been making an impact on society since at least the 1960s. AI has long suffered from every time they come up with something new it gets renamed and becomes important (where it makes sense) without giving AI credit.
From what I can tell most in AI are currently hoping LLMs reach that point quick just because the hype is not helping AI at all.
Workaccount2 · 4h ago
Yesterday my dad, in his late 70's, used Gemini with a video stream to program the thermostat. He then called me to tell me this, rather then call me to come stop by and program the thermostat.
You can call this hype, maybe it is all hype until LLMs can work on 10M LOC codebases, but recognize that LLMs are a shift that is totally incomparable to any previous AI advancement.
ferguess_k · 2h ago
Yeah. As a mediocre programmer I'm really scared about this. I don't think we are very far from AI replacing the mediocre programmers. Maybe a decade, at most.
I'd definitely like to improve my skills, but to be realistic, most of the programmers are not top-notch.
orochimaaru · 3h ago
That is what open ai’s non-profit economic research arm has claimed. LLMs will fundamentally change how we interact with the world like the Internet did. It will take time like the Internet and a couple of hype cycle pops but it will change the way we do things.
It will help a single human do more in a white collar world.
> He then called me to tell me this, rather then call me to come stop by and program the thermostat.
Sounds like AI robbed you of an opportunity to spend some time with your Dad, to me
jabits · 1h ago
Or maybe instead of spending time with your dad on a bs menial task, you could spent time fishing with him…
bluefirebrand · 1h ago
It's nice to think that but life and relationships are also composed of the little moments, which sometimes happen when someone asks you over to help with a "bs menial task"
It takes five minutes to program the thermostat, then you can have a beer on the patio if that's your speed and catch up for a bit
Life is little moments, not always the big commitments like taking a day to go fishing
That's the point of automating all of ourselves out of work, right? So we have more time to enjoy spending time with the people we love?
So isn't it kind of sad if we wind up automating those moments out of our lives instead?
bluGill · 3h ago
There are clearly a lot of useful things about LLMs. However there is a lot of hype as well. It will take time to separate the two.
danielbln · 4h ago
Bitter lesson applies here as well though. Generalized models will beat specialized models given enough time and compute. How much bespoke NLP is there anymore? Generalized foundational models will subsume all of it eventually.
johnecheck · 4h ago
You misunderstand the bitter lesson.
It's not about specialized vs generalized models - it's about how models are trained. The chess engine that beat Kasparov is a specialized model (it only plays chess), yet it's the bitter lesson's example for the smarter way to do AI.
Chess engines are better at chess than LLMs. It's not close. Perhaps eventually a superintelligence will surpass the engines, but that's far from assured.
Specialized AI are hardly obsolete and may never be. This hypothetical superintelligence may even decide not to waste resources trying to surpass the chess AI and instead use it as a tool.
ses1984 · 4h ago
Generalized models might be better but they are rarely more efficient.
ferguess_k · 5h ago
Yeah I agree with it. There is a lot of hype, but there is some potentials there.
BolexNOLA · 5h ago
Yeah “AI” tools (such a loose term but largely applicable) have been involved in audio production for a very long time. They have actually made huge strides with noise removal/voice isolation, auto transcription/captioning, and “enhancement” in the last five years in particular.
I hate Adobe, I don’t like to give them credit for anything. But their audio enhance tool is actual sorcery. Every competitor isn’t even close. You can take garbage zoom audio and make it sound like it was borderline recorded in a treated room/studio. I’ve been in production for almost 15 years and it would take me half a day or more of tweaking a voice track with multiple tools that cost me hundreds of dollars to get it 50% as good as what they accomplish in a minute with the click of a button.
nightski · 3h ago
Saying we don't "need" AGI is like saying we don't need electricity. Sure life existed before we had that capability, but it would be very transformative. Of course we can make specialized tools in the mean time.
hoosieree · 58m ago
The error in this argument is that electricity is real.
charcircuit · 3h ago
Can you give an example how it would be transformative compared to specialized AI?
Jensson · 2h ago
AGI is transformative in that it lets us replace knowledge workers completely, specialized AI requires knowledge workers to train them for new tasks while AGI doesn't.
fennecfoxy · 2h ago
Because it could very well exceed our capabilities beyond our wildest imaginations.
Because we evolved to get where we are, humans have all sorts of messy behaviours that aren't really compatible with a utopian society. Theft, violence, crime, greed - it's all completely unnecessary and yet most of us can't bring ourselves to solve these problems. And plenty are happy to live apathetically while billionaires become trillionaires...for what exactly? There's a whole industry of hyper-luxury goods now, because they make so much money even regular luxury is too cheap.
If we can produce AGI that exceeds the capabilities of our species, then my hope is that rather than the typical outcome of "they kill us all", that they will simply keep us in line. They will babysit us. They will force us all to get along, to ensure that we treat each other fairly.
As a parent teaches children to share by forcing them to break the cookie in half, perhaps AI will do the same for us.
davidivadavid · 1h ago
Oh great, can't wait for our AI overlords to control us more! That's definitely compatible with a "utopian society"*.
Funnily enough, I still think some of the most interesting semi-recent writing on utopia was done ~15 years ago by... Eliezer Yudkowsky. You might be interested in the article on "Amputation of Destiny."
What if AGI is just a bunch of specialized AIs put together?
It would seem our own generalized intelligence is an emergent property of many, _many_ specialized processes
I wonder if AI is the same
Jensson · 2h ago
> It would seem our own generalized intelligence is an emergent property of many, _many_ specialized processes
You can say that about other animals, but about humans it is not so sure. No animal can be taught as general set of skills as a human can, they might have some better specialized skills but clearly there is something special that makes humans so much more versatile.
So it seems there was this simple little thing humans got that makes them general, while for example our very close relatives the monkeys are not.
fennecfoxy · 2h ago
Humans are the ceiling at the moment yes, but that doesn't mean the ceiling isn't higher.
Science is full of theories that are correct per our current knowledge and then subsequently disproven when research/methods/etc improves.
Humans aren't special, we are made from blood & bone, not magic. We will eventually build AGI if we keep at it. However unlike VCs with no real skills except having a lot of money™, I couldn't say whether this is gonna happen in 2 years or 2000.
Jensson · 1h ago
Question was if cobbling together enough special intelligence creates general intelligence. Monkeys has a lot of special intelligence that our current AI models can't come close to, but still aren't seen as general intelligence like humans, so there is some little bit humans has that isn't just another special intelligence.
mike_ivanov · 2h ago
It may be a property of (not only of?) humans that we can generate specialized inner processes. The hardcoded ones stay, the emergent ones come and go. Intelligence itself might be the ability to breed new specialized mental processes on demand.
AndrewKemendo · 4h ago
This debate is exhausting because there's no coherent definition of AGI that people agree on.
I made a google form question for collecting AGI definitions cause I don't see anyone else doing it and I find it infinitely frustrating the range of definitions for this concept:
There are generational policy and societal shifts that need to be addressed somewhere around true Competent AGI (50% of knowledge work tasks automatable). Just like climate change, we need a shared lexicon to refer to this continuum. You can argue for different values of X but the crucial point is if X% of knowledge work is automated within a decade, then there are obvious risks we need to think about.
So much of the discourse is stuck at “we will never get to X=99” when we could agree to disagree on that and move on to considering the x=25 case. Or predict our timelines for X and then actually be held accountable for our falsifiable predictions, instead of the current vide based discussions.
bigyabai · 4h ago
It is a marketing term. That's it. Trying to exhaustively define what AGI is or could be is like trying to explain what a Happy Meal is. At it's core, the Happy Meal was not invented to revolutionize food eating. It puts an attractive label on some mediocre food, a title that exists for the purpose of advertisement.
There is no point collecting definitions for AGI, it was not conceived as a description for something novel or provably existent. It is "Happy Meal marketing" but aimed for adults.
HarHarVeryFunny · 45m ago
The name AGI (i.e. generalist AI) was originally intended to contrast with narrow AI which is only capable of one, or a few, specific narrow skills. A narrow AI might be able to play chess, or distinguish 20 breeds of dog, but wouldn't be able to play tic tac toe because it wasn't built for that. AGI would be able to learn to do anything, within reason.
The term AGI is obviously used very loosely with little agreement to it's precise definition, but I think a lot of people take it to mean not only generality, but specifically human-level generality, and human-level ability to learn from experience and solve problems.
A large part of the problem with AGI being poorly defined is that intelligence itself is poorly defined. Even if we choose to define AGI as meaning human-level intelligence, what does THAT mean? I think there is a simple reductionist definition of intelligence (as the word is used to refer to human/animal intelligence), but ultimately the meaning of words are derived from their usage, and the word "intelligence" is used in 100 different ways ...
AndrewKemendo · 55m ago
That’s historically inaccurate
My masters thesis advisor Ben Goertzel popularized the term and has been hosting the AGI conference since 2008:
I had lunch with Yoshua Bengio at AGI 2014 and it was most of the conversation that day
vonneumannstan · 3h ago
Is this supposed to be a gotcha? We know these systems are typically trained using RL and they are exceedingly good at learning games...
johnb231 · 2m ago
No it is not a “gotcha” and I don’t understand how you got that impression.
throw_nbvc1234 · 7h ago
This sounds like a problem that could be solved around the corner with a caveat.
Games generally are solvable for AI because they have feedback loops and a clear success or failure criteria. If the "picking up a Joystick" part is the limiting factor, sure. But why would we want robots to use an interface (especially a modern controller) heavily optimized for human hands; that seems like the definition of a horseless carriage.
I'm sure if you compared a monkey and a dolphins performance using a joystick you'd get results that aren't really correlated with their intelligence. I would guess that if you gave robots an R2D2 like port to jack into and play a game, that problem could be solved relatively quickly.
xnickb · 7h ago
Just like OpenAI early on promised us an AGI and showed us how it "solved" Dota 2.
They also claimed it "learned" to play by playing itself only however it was clear that most of the advanced techniques were borrowed from existing AI and by observing humans.
No surprise they gave up on that project completely and I doubt they'll ever engage in anything like that again.
Money better spent on different marketing platforms.
jsheard · 6h ago
It also wasn't even remotely close to learning Dota 2 proper. They ran a massively simplified version of the game where the AI and humans alternated between playing one of two pre-defined team compositions, meaning >90% of the games characters and >99.999999% of the possible compositions and matchups weren't even on the table, plus other standard mechanics were also changed or disabled altogether for the sake of the AI team.
Saying you've solved Dota after stripping out nearly all of its complexity is like saying you've solved Chess, but on a version where the back row is all Bishops.
xnickb · 6h ago
Exactly. What I find surprising in this story though is not the OpenAI. It's investors not seeing through these blatant.. lets call them exaggerations of the reality and still trusting the company with their money. I know I wouldn't have. But then again, maybe that's why I'm poor.
ryandrake · 5h ago
In their hearts, startup investors are like Agent Mulder: they Want To Believe. Especially after they’ve already invested a little. They are willing to overlook obvious exaggerations up to and including fraud, because the alternative is admitting their judgment is not sound.
Look at how long Theranos went on! Miraculous product. Attractive young founder with all the right pedigree, credentials, and contacts, dressed in black trurtlenecks. Hell, she even talked like Steve Jobs! Investors never had a chance.
jdross · 5h ago
They already have 400 million daily users and a billion people using the product, with billions of consumer subscription revenue, faster than any company ever. They are also aggregating R&D talent at a density never before seen in Silicon Valley
That is what investors see. You seem to treat this as a purity contest where you define purity
zaphar · 5h ago
Also apparently still not making a profit.
No comments yet
xnickb · 5h ago
I'm speaking about past events. Perhaps I didn't make it clear enough
rowanG077 · 4h ago
I agree that restricting the hero pool is a huge simplification. But they did play full 5v5 standard dota with just a restricted hero pool of 17 heroes and no illusions/control units according to theverge (https://www.theverge.com/2019/4/13/18309459/openai-five-dota...). It destroyed the professionals.
As an ex dota player, I don't think this is that far off from having full on, all heroes dota. Certainly not as far of as you are making it sound.
And dota is one of the most complex games, I expect for example that an AI would instantly solve CS since aim is such a large part of the game.
Jensson · 3h ago
> It destroyed the professionals.
Only the first time, later when it played better players it always lost. Players learned the faults of the AI after some time in game and the AI had very bad late game so they always won later.
rowanG077 · 3h ago
Not on the last iteration.
mistercheph · 3h ago
Another issue with the approach is that the model had direct access to game data, that is simply an unfair competitive advantage in dota, and it is obvious why that advantage would be unfair in CS.
It is certainly possible, but
i won't be impressed by anything "playing CS" that isn't running a vision model on a display and moving a mouse, because that is the game. The game is not abstractly reacting to enemy positions and relocating the cursor, it's looking at a screen, seeing where the baddy is and then using this interface (the mouse) to get the cursor there as quickly as possible.
It would be like letting an AI plot its position on the field and what action its taking during a football match and then saying "Look, The AI would have scored dozens of times in this simulation, it is the greatest soccer player in the world!" No, sorry, the game actually requires you to locomote, abstractly describing your position may be fun but it's not the game
rowanG077 · 3h ago
Did you read the paper? It had access to the dota 2 bot API, which is some gamestate but very far from all gamestate. It also had artifially limited reaction to something like 220ms, worse then professional gamers.
But then again, that is precisely the point. A chess bot also has access to gigabytes of perfect working memory. I don't see people complaining about that. It's perfectly valid to judge the best an AI can do vs the best a human can do. It's not really fair to take away exactly what a computer is good at from an AI and then say: "Look but the AI is now worse". Else you would also have to do it the other way around. How well could a human play dota if it only had access to the bot API. I don't think they would do well at all.
scotty79 · 6h ago
It was 6 years ago. I'm sure now there'd be no contest now if OpenAI dedicated resources to it, which it won't because it's busy with solving entirety of human language before others eat their lunch.
spektral23 · 5h ago
Funnily enough, even dota2 has grown much more complex than it was 6 years ago, so it's a harder problem to solve today than it was back then
xnickb · 6h ago
What do you base your certainty on? Were there any significant enough breakthroughs in the AGI?
scotty79 · 6h ago
ARC-AGI, while imagined as super hard for AI, was beaten enough that they had to come up with ARC-AGI-2.
hbsbsbsndk · 5h ago
"AI tend to be brittle and optimized for specific tasks, so we made a new specific task and then someone optimized for it" isn't some kind of gotcha. Once ARC puzzles became a benchmark they ceased to be meaningful WRT "AGI".
scotty79 · 37m ago
So if DOTA became a benchmark same way Chess or Go became earlier it would be promptly beaten. It just didn't stick before people moved to more useful "games".
fennecfoxy · 1h ago
To be fair humans have had quite a few million years across a growing population to gather all of the knowledge that we have.
As we're learning with LLMs, the dataset is what matters - and what's awesome is that you can see that in us, as well! I've read that our evolution is comparatively slow to the rate of knowledge accumulation in the information age - and that what this means is that you can essentially take a caveman, raise them in our modern environment and they'll be just as intelligent as the average human today.
But the core of our intelligence is logic/problem solving. We just have to solve higher order problems today, like figuring out how to make that chart in excel do the thing you want, but in days past it was figuring out how to keep the fire lit when it's raining. When you look at it, we've possessed the very core of that problem solving ability for quite a while now. I think that is the key to why we are human, and our close ancestors monkeys are...still just monkeys.
It's that problem solving ability that we need to figure out how to produce within ML models, then we'll be cooking with gas!
mellosouls · 7h ago
The point isn't about learning video games its about learning tasks unrelated to its specific competency generally.
jandrese · 2h ago
> But why would we want robots to use an interface (especially a modern controller) heavily optimized for human hands; that seems like the definition of a horseless carriage.
Elon's response to this is that if we want these androids to replace human jobs then the lowest friction alternative is for the android to be able to do anything a human can do in a human amount of space. A specialized machine is faster and more efficient, but comes with engineering and integration costs that create a barrier to entry. Elon learned this lesson the hard way when he was building out the gigafactories and ended up having to hire a lot of people to do the work while they sorted out the issues with the robots. To someone like Elon a payroll is an ever growing parasite on a companies bottom line, far better if the entire thing is automated.
jappgar · 6h ago
A human would learn it faster, and could immediately teach other humans.
AI clearly isn't at human level and it's OK to admit it.
johnb231 · 7h ago
No, the joystick part is really not the limiting factor. They’ve already done this with a direct software interface. Physical interface is a new challenge. But overall you are missing the point.
suddenlybananas · 9h ago
It's because humans (and other animals) have enormous innate capacities and knowledge which makes learning new things much much simpler than if you start from scratch. It's not really because of human's computational capacity.
xnx · 4h ago
> enormous innate capacities and knowledge
Hundreds of millions of years of trial-and-error biological pre-training where survival/propagation is the reward function
Nopoint2 · 3h ago
There is just no reason to believe that we are born with some insanely big library of knowledge, and it sounds completely impossible. How would it be stored, and how would we even evolve it?
It just isn't needed. Just like you can find let's say kangaroos in the latent space of an image generator, so we learn abstract concepts and principles of how things work as a bonus of learning to process the senses.
Maybe a way to AGI could be figuring out how to combine a video generator with a LLM or something similar in a way that allows it to understand things intuitively, instead of doing just lots and lots of some statistical bullsit.
Jensson · 3h ago
> There is just no reason to believe that we are born with some insanely big library of knowledge, and it sounds completely impossible. How would it be stored, and how would we even evolve it?
We do have that, ever felt fear of heights? That isn't learned, we are born with it. Same with fear of small moving objects like spiders or snakes.
Such things are learned/stored very different from memories, but its certainly there and we can see animals also have those. Like cats gets very scared of objects that are long and appear suddenly, like a cucumber, since their genetic instincts thinks its a snake.
throwup238 · 2h ago
> Like cats gets very scared of objects that are long and appear suddenly, like a cucumber, since their genetic instincts thinks its a snake.
After having raised four dozen kittens that a couple of feral sisters gave birth to in my garage, I’m certain that is nonsense. It’s an internet meme that became urban legend.
I don’t think they have ever even reacted to a cucumber, and I have run many experiments because my childhood cat loved cucumbers (we’d have to guard the basket of cucumbers after harvest, otherwise she’d bite every single one of them… just once).
Nopoint2 · 2h ago
Of course it is learned, and fear is triggered by anything unfamiliar, that causes a high reconstruction error. Because it means you don't understand it, and it could be dangerous. We are just not used to encoding anything so deep below the eye level, and it freaks us out.
Jensson · 2h ago
Do you really think every single ant is learning all that on its own? And if ants can store that in their DNA, why don't you think other animals can? DNA works just fine as generic information storage, there are obviously a ton of behaviors and information encoded there from hundreds of millions of years of survival of the fittest.
MrScruff · 8h ago
By innate do you mean evolved/instinctive? Surely even evolved behaviour must be expressed as brain function, and therefore would need a brain capable of handling that level of processing.
I don't think it's clear how much of a human brains function exists at birth though, I know it's theorised than even much of the sensory processing has to be learned.
suddenlybananas · 8h ago
I'm not arguing against computational theory of mind, I'm just saying that innate behaviours don't require the same level of scale as learnt ones.
Existing at birth is not the same thing as innate. Puberty is innate but it is not present at birth.
MrScruff · 8h ago
That's an interesting point. I can see that, as you say puberty and hormones impact brain function and hence behaviour, and those are inate and not learned. But at least superfically that would appear to be primarily broad behavioural effects, similar to what might be induced by medication. Rather than something that impacts pure abstract problem solving, which I guess is what the Atari games are supposed to represent?
rafaelmn · 7h ago
This is obviously wrong from genetic defects that cause predictable development problems in specialized areas. They are innate but not present at birth.
nlitened · 9h ago
> the human brain, of which we don't have a clear understanding of the compute capacity
Neurons have finite (very low) speed of signal transfer, so just by measuring cognitive reaction time we can deduce upper bounds on how many _consecutive_ neuron connections are involved in reception, cognitive processing, and resulting reaction via muscles, even for very complex cognitive processes. And the number is just around 100 consecutive neurons involved one after another. So “the algorithm” could not be _that_ complex in the end (100x matmul+tanh?)
Granted, a lot of parallelism and feedback loops are involved, but overall it gives me (and many others) an impression that when the AGI algorithm is ever found, it’s “mini” version should be able to run on modest 2025 hardware in real time.
johnb231 · 8h ago
> (100x matmul+tanh?)
Biological neurons are way more complex than that. A single neuron has dentritic trees with subunits doing their own local computations. There are temporal dynamics in the firing sequences. There is so much more complexity in the biological networks. It's not comparable.
woolion · 7h ago
You could implement a Turing-machine with humans acting physically operating as logic gates. Then, every human is just a boolean function.
Jensson · 3h ago
Neurons are stateful though, it is core to their function and how they learn.
neffy · 7h ago
This is exactly it. Biology is making massive use of hacked real time local network communication in ways we haven´t begun to explore.
scajanus · 8h ago
The granted is doing a lot of work there. In fact, if you imagine a computer being able to do similar tasks as human brain can in around 100 steps, it becomes clear that considering parallelism is absolutely critical.
Actually no, it's not interesting at all. Vague dismissal of an outsider is a pretty standard response by insecure academic types. It could have been interesting and/or helpful to the conversation if they went into specifics or explained anything at all. Since none of that's provided, it's "OpenAI insider" vs John Carmack AND Richard Sutton. I know who I would bet on.
handsclean · 2h ago
It seems that you’ve only read the first part of the message. X sometimes aggressively truncates content with no indication it’s done so. I’m not sure this is complete, but I’ve recovered this much:
> I read through these slides and felt like I was transported back to 2018.
> Having been in this spot years ago, thinking about what John & team are thinking about, I can't help but feel like they will learn the same lesson I did the hard way.
> The lesson: on a fundamental level, solutions to these games are low-dimensional. No matter how hard you hit them with from-scratch training, tiny models will work about as well as big ones. Why? Because there's just not that many bits to learn.
> If there's not that many bits to learn, then researcher input becomes non-negligible.
> "I found a trick that makes score go up!" -- yeah, you just hard-coded 100+ bits of information; a winning solution is probably only like 1000 bits. You see progress, but it's not the AI's.
> In this simplified RL setting, you don't see anything close to general intelligence. The neural networks aren't even that important.
> You won't see _real_ learning until you absorb a ton of bits into the model. The only way I really know to do this is with generative modeling.
> A classic example: why is frame stacking just as good as RNNs? John mentioned this in his slides. Shouldn't a better, more general architecture work better?
> YES, it should! But it doesn't, because these environments don't heavily encourage real intelligence.
ActivePattern · 5h ago
It's a OpenAI researcher that's worked on some of their most successful projects, and I think the criticism in his X thread is very clear.
Systems that can learn to play Atari efficiently are exploiting the fact that the solutions to each game are simple to encode (compared to real world problems). Furthermore, you can nudge them towards those solutions using tricks that don't generalize to the real world.
dgb23 · 2h ago
That sounds like an extremely useful insight that makes this kind of research even more valuable.
lairv · 4h ago
Alex Nichol worked on "Gotta Learn Fast" in 2018 which Carmack mentions in his talk, he also worked on foundational deep learning methods like CLIP, DDPM, GLIDE, etc. Reducing him to a "seething openai insider" seems a bit unfair
kadushka · 5h ago
He did go into specifics and explained his point. Or have you only read his first post?
quadrature · 3h ago
Do you have an X account, if you're not logged in you'll only see the first post in the thread.
threatripper · 3h ago
x.com/... -> xcancel.com/...
MattRix · 3h ago
It’s not vague, did you only see the first tweet or the entire thread?
lancekey · 17m ago
I think some replies here are reading the full twitter thread, while others (not logged in?) see only the first tweet. The first tweet alone does come off as a dismissal with no insight.
jjulius · 5h ago
I appreciate how they don't tell us what lesson they learned.
dcre · 4h ago
It is a thread. You may have only seen the first tweet because Twitter is a user-hostile trash fire.
“The lesson: on a fundamental level, solutions to these games are low-dimensional. No matter how hard you hit them with from-scratch training, tiny models will work about as well as big ones. Why? Because there's just not that many bits to learn.”
Thank you for clarifying. I don't have a Twitter account, and the linked tweet genuinely looks like a standalone object. Mea culpa.
zeroq · 6h ago
>> "they will learn the same lesson I did"
Which is what? Don't trust Altman? x)
andy_ppp · 7h ago
My bet is on Carmack.
WithinReason · 4h ago
"Graphics Carmack" is a genius but that doesn't mean that "AI Carmack" is too.
MrLeap · 4h ago
I wouldn't bet against him. "The Bitter Lesson" may imply an advantage to someone who historically has been at the tip of the spear for squeezing the most juice out of GPU hosted parallel computation.
Graphics rendering and AI live on the same pyramid of technology. A pyramid with a lot of bricks with the initials "JC" carved into them, as it turns out.
kadushka · 3h ago
Only if computation is the bottleneck. GPT-4.5 shows it’s not.
cheschire · 1h ago
Carmack is always a genius, but like most people he requires luck, and like most people, the house always wins. Poor Armadillo Aerospace.
ramesh31 · 4h ago
What has he shipped in the last 20 years? Oculus is one thing, but that was firmly within his wheelhouse of graphics optimization. Abrash and co. handled the hardware side of things.
Carmack is a genius no doubt. But genius is the result of intense focused practice above and beyond anyone else in a particular area. Trying
to extend that to other domains has been the downfall of so many others like him.
No comments yet
speed_spread · 6h ago
I suspect Carmack in the Dancehall with the BFG.
alexey-salmin · 2h ago
Each of these games is low-dimensional and require not the "intelligence" but more like "reflexes", I tend to agree.
However making a system that can beat an unknown game does require generalization. If not real a intelligence (whatever that means) but at the level of say "a wolf".
Whether it can arise from RL alone is not certain, but it's there somewhere.
cmiles74 · 6h ago
From a marketing perspective, this strikes me as a very predictable response.
And I say this while most certainly not being as knowledgeable as this openai insider. So it even I can see this, then it's kinda bad, isn't it?
fmbb · 7h ago
Can you explain which parts you think are bad and why?
jjulius · 5h ago
Right? "Even I can see this" isn't exactly enlightening.
koolala · 8h ago
I wish he did this with VR environment instead like they mention at the start of the slides. A VR environment with a JPEG camera filter, physics sim, noise, robot simulation. If anyone could program that well its him.
Using real life robots is going to be a huge bottleneck for training hours no matter what they do.
ploden · 2h ago
Why would AGI choose to be embodied? We talk about creating a superior intelligence and having it drive our cars and clean our homes. The scenario in Dan Simmons' Hyperion seems much more plausible: we invent AGI and it disappears into the cloud and largely ignores us.
jwmcq · 9m ago
Looking at other examples in sci-fi, perhaps to stop my body from pressing its off-switch?
fusionadvocate · 1h ago
It doesn't need to be permanent. If humans could escape from their embodiment temporarily they would certainly do so. Being permanently bounded to a physical interface is definitely a disadvantage.
kamranjon · 12h ago
I was really excited when I heard Carmack was focusing on AI and am really looking forward to watching this when the video is up - but just from looking at the slides it seems like he tried to build a system that can play the Atari? Seems like a fun project, but curious what will come out of it or if there is an associated paper being released.
johnb231 · 12h ago
Atari games are widely used in Reinforcement Learning (RL) research as a standard benchmark.
The goal is to develop algorithms that generalize to other tasks.
sigmoid10 · 11h ago
They were highly used. OpenAI even included them in their RL Gym library back in the old days when they were still doing open research. But if you look at this leaderboard from 7 (yes, seven!) years ago [1], most of them were already solved way beyond human capabilities. But we didn't get a really useful general purpose algorithm out of it. As an AI researcher, I always considered Atari a fun academic exercise, but nothing more. Similar to how recognising characters using convnets was cool in the nineties and early 00s, but didn't give us general purpose image understanding. Only modern GPUs and massive training datasets did. Nowadays most cutting-edge RL game research focuses on much more advanced games like Minecraft which is thought to be better suited. But I'm pretty sure it's still not enough. Even role-playing GTA VI won't be. We probably need a pretty advanced physical simulation of the real world before we can get agents to handle the real world. But that means solving the problem of generating such an environment first, because you can't train on the actual real world due to the sample inefficiency of all current algorithms. Nvidia is doing some really interesting research in this direction by combining physics simulation and image generation models to simulate an environment, while getting accuracy and diversity at the same time into training data. But it still feels like some key ingredient is missing.
I watched the talk live. I felt that his main argument was that Atari _looks_ solved, but there's still plenty of value that could be gained by revisiting these "solved" games. For one, learning how to play games through a physical interface is a way to start engaging with the kinds of problems that make robotics hard (e.g., latency). They're also a good environment to study catastrophic forgetting: an hour of training on one game shouldn't erase a model's ability to play other games.
I think we could eventually saturate Atari, but for now it looks like it's still a good source of problems that are just out of reach of current methods.
koolala · 3h ago
Is a highly specialized bespoke robot for a Atari controller really that different? If anyone cared about latency they could have added it to the emulated controls and video with random noise.
gregdeon · 2h ago
I think it is. Latency was just one of the problems he described. A physical controller sometimes adds "phantom inputs" as the joystick transitions between two inputs. Physical actuators also slow down with wear. A physical Atari-playing robot needs to learn qualitatively different strategies that are somewhat more robust to these problems. Emulators also let the bot take as much time as it needs between frames, which is much easier than playing in real time. To me, all of this makes a physical robot seem like a decent way to start engaging with problems that come up in robotics but not simulated games.
mschuster91 · 10h ago
> But it still feels like some key ingredient is missing.
Continuous training is the key ingredient. Humans can use existing knowledge and apply it to new scenarios, and so can most AI. But AI cannot permanently remember the result of its actions in the real world, and so its body of knowledge cannot expand.
Take a toddler and an oven. The toddler has no concept of what an oven is other than maybe that it smells nice. The toddler will touch the oven, notice that it experiences pain (because the oven is hot) and learn that oven = danger. Place a current AI in a droid toddler body? It will never learn and keep touching the oven as soon as the information of "oven = danger" is out of the context window.
For some cases this inability to learn is actually desirable. You don't want anyone and everyone to be able to train ChatGPT unsupervised, otherwise you get 4chan flooding it with offensive crap like they did to Tay [1], but for AI that physically interacts with the meatspace, constant evaluation and learning is all but mandatory if it is to safely interact with its surroundings. "Dumb" robots run regular calibration cycles for their limbs to make sure they are still aligned to compensate for random deviations, and so will AI robots.
This kind of context management is not that hard, even when building LLMs. Especially when you have huge windows like we do today. Look at how ChatGPT can remember things permanently after you said them once using a function call to edit the permanent memory section inside the context. You can also see that in Anthropic's latest post on Claude 4 where it learns to play Pokemon. The only remaining issue here is maybe how to diffuse explicit knowledge from the stored context into the weights. Andrej Karpathy wrote a good piece on this recently. But personally I believe this might not even be necessary if you can manage your context well enough and see it more like RAM while the LLM is the CPU. For your example you can then always just fetch such information from a permanent storage like a VDB and load it into context once you enter an area in the real world.
mr_toad · 7h ago
Big context windows are a poor substitute for updating the weights. Its like keeping a journal because your memory is failing.
fzzzy · 5h ago
It reminds me of the movie Memento.
vectorisedkzk · 9h ago
Having used vectorDBs before, we're very much not there yet. We don't have any appreciable amounts of context for any reasonable real-life memory. It works if that is the most recent thing you did. Have you talked to an LLM for a day? Stuff is gone before the first hour. You have to use every trick currently in the book, treat context like it's your precious pet
sigmoid10 · 9h ago
VectorDBs are basically just one excuse of many to make up for a part of the system that is lacking capability due to technical limitations. I'm currently at 50:50 if the problems will be overcome directly by the models or by such support systems. Used to be 80:20 but models have grown in usefulness much faster than all the tools we built around them.
mschuster91 · 9h ago
> This kind of context management is not that hard, even when building LLMs.
It is, at least if you wish to be in the meatspace, that's my point. Every day has 86400 seconds during which a human brain constantly adapts to and learns from external input - either directly as it's being awake or indirectly during nighttime cleanup processes.
On top of that, humans have built-in filters for training. Basically, we see some drunkard shouting about the Hollow Earth on the sidewalk... our brain knows that this is a drunkard and that Hollow Earth is absolutely crackpot material, so if it stores anything at all then the fact that there is a drunkard on that street and one might take another route next time, but the drunkard's rambling is forgotten maybe five minutes later.
AI, in contrast, needs to be hand-held by humans during training that annotate, "grade" or weigh information during the compilation of the training dataset, in order that the AI knows what is written in "Mein Kampf" so it can answer questions upon it, but that it also knows (or at least: won't openly regurgitate) that the solution to economic problems isn't to just deport Jews.
And huge context windows aren't the answer either. My wife says me, she would like to have a fruit cake for her next birthday. I'll probably remember that piece of information (or at the very least I'll write it down)... but an AI butler? I'd be really surprised if this is still in its context space in a year, and even if it is, I would not be surprised if it weren't able to recall that fact.
And the final thing is prompts... also not the answer. We've seen it just a few days ago with Grok - someone messed with the system prompt so it randomly interjected "white genocide" claims into completely unrelated conversation [1] despite hopefully being trained on a ... more civilised dataset, and to the contrary, we've also seen Grok reply to Twitter questions in a way that suggest that it is aware its training data is biased.
>Every day has 86400 seconds during which a human brain constantly adapts to and learns from external
That's not even remotely true. At least not in the sense that it is for context in transformer models. Or can you tell me all the visual and auditory inputs you experienced yesterday at the 45232nd second? You only learn permanently and effectively from particular stimulation coupled with surprise. That has a sample rate which is orders of magnitude lower. And it's exactly the kind of sampling that can be replicated with a run-of-the-mill persistent memory system
for an LLM. I would wager that you could fit most people's core experiences and memories that they can randomly access at any moment into a 1000 page book - something that fits well into state of the art context windows. For deeper more detailed things you can always fall back to another system.
bluesroo · 4h ago
Your definition of "learning" is incomplete because you're applying LLM concepts to how human brains work. An LLM only "learns" during training. From that point forward all it has is its context and vector DBs. If an LLM and vector DB is not actively interacted with, nothing happens to it. However for the brain, experiencing IS learning. And the brain NEVER stops experiencing.
Just because I don't remember my experiences at second 45232 on May 22, doesn't mean that my brain was not actively adapting to my experiences at that moment. The brain does a lot more learning than just what is conscious. And then when I went to sleep the brain continued pruning and organizing my unconscious learning for the day.
Seeing if someone can go from token to freeform physical usefulness will be interesting. I'm of the belief that LLMs are too verbose and energy intensive to go from language regurgitation machines to moving in the real world according to free form prompting. It may be accomplishable with the vast amount of hype investment, but I think the energy requirements and latency will make an LLM-based approach economically infeasible.
ewoodrich · 3h ago
> You only learn permanently and effectively from particular stimulation coupled with surprise.
This is just, not true. A single 2min conversation with emotional or intellectual resonance can significantly alter a human’s thought process for years. There are some topics where every time they come up directly or analogously I can recall something a teacher told me in high school that “stuck” with me for whatever reason. And it isn’t even a “core” experience, just something that instantly clicked for my brain and altered my problem solving. At the time, there’s no heuristic that could predict how or why that particular interaction should have that kind of staying power.
Not to mention, experiences that subtly alter thinking or behavior just by virtue of providing some baseline familiarity instead of blank slate problem solving or routine. Like how you subtly adjust how you interact with coworkers based on the culture of your current company over time vs the last without any “flash” of insight required.
epolanski · 8h ago
> Humans can use existing knowledge and apply it to new scenarios, and so can most AI
Doesn't the article states that this is not true? AI cannot apply to B what it learned about A.
mschuster91 · 7h ago
Well, ChatGPT knows about the 90s Balkan wars, a topic to which LWT hasn't made an episode that I'm aware of, and yet I can ask it to write a script for a Balkan wars episode that reads surprisingly like John Oliver while being reasonably correct.
epolanski · 7h ago
Essentially Carmack pointed in the slides that teaching AI to play game a, b or c didn't improve AI at all at learning game d from scratch.
That's essentially what we're looking for when we talk about general intelligence, the capability to adapting what we know to what we know nothing about.
aatd86 · 10h ago
it's more than that. Our understanding from space and time could be stemming from continuous training.
Every time we look at something, there seems to be a background process that is categorizing items that are on the retinal image.
This is a continuous process.
newsclues · 8h ago
Being highly used in the past is good, it's a benchmark to compare against.
tschillaci · 10h ago
You will find many agents that solved (e.g., finished, reached high score) atari games, but there is still so much more work to do in the field. I wrote my Master's thesis on how to learn from few interactions with the game, so that if the algorithm is ported to actual robots they don't need to walk and fall for centuries before learning behaviors. I think there is more research to do on higher levels of generalization: when you know how to play a few video games, you quickly understand how to play a new one intuitively, and I haven't seen thorough research on that.
lo0dot0 · 4h ago
I can tell you right now without any research that video game designers reuse interface patterns and game mechanics that were already known when making new games. Those patterns and mechanics are also often analogies for real life allowing humans to intuitively play the games. If people can't play your game intuitively, they might say it's a bad game.
Jensson · 2h ago
So why can't AI learn those and reapply the same understanding to new games?
albertzeyer · 9h ago
His goal was not just to solve Atari games. That was already done.
His goal is to develop generic methods. So you could work with more complex games or the physical world for that, as that is what you want in the end. However, his insight is, you can even modify the Atari setting to test this, e.g. to work in realtime, and the added complexity by more complex games doesn't really give you any new additional insights at this point.
mike_hearn · 2h ago
But how is this different to what NVIDIA have already done? They have robots that can achieve arbitrary and fluid actions in the real world by training NNs in very accurate GPU simulated environments using physics engines. Moving a little Atari stick around seems like not much compared to sorting through your groceries etc.
The approach NVIDIA are using (and other labs) clearly works. It's not going to be more than a year or two now before robotics is as solved as NLP and chatbots are today.
modeless · 11h ago
He says they will open source it which is cool. I agree that I don't understand what's novel here. Playing with a physical controller and camera on a laptop GPU in real time is cool, and maybe that hasn't specifically been done before, but it doesn't seem surprising that it is possible.
If it is substantially more sample efficient, or generalizable, than prior work then that would be exciting. But I'm not sure if it is?
RetroTechie · 6h ago
Maybe that's exactly his goal: not to come up with something that beats the competition, but play with the architecture, get a feel for what works & what doesn't, how various aspects affect the output, and improve on that. Design more efficient architectures, or come up with something that has unique features compared to other models.
If so, scaling up may be more of a distraction rather than helpful (besides wasting resources).
I hope he succeeds in whatever he's aiming for.
cryptoz · 11h ago
DeepMind’s original demos were also of Atari gameplay.
gadders · 7h ago
I want smarter NPCs in games.
andy_ppp · 7h ago
I still don’t think we have a clear enough idea of what a concept is to be able to think about AGI. And then being able to use concepts from one area to translate into another area, what is the process by which the brain combines and abstracts ideas into something new?
throw310822 · 6h ago
Known entities are recurring patterns (we give names to things that occur more than once, in the world or in our thoughts). Concepts are recurring thought patterns. Abstractions, relations, metaphors, are all ways of finding and transferring patterns from one domain to another.
andy_ppp · 6h ago
Sure, I understand what the terminology means but I don't believe we get to AGI without some ability to translate the learning of say using a mouse to using a trackpad in a simple way. Humans make these translations all the while, you know how to use a new room and the items in it automatically but I personally see the systems we have built are currently very brittle when they see new environments because they can't simplify everything to its fundamentals and then extrapolate back to more complex tasks. You could train a human on using an Android phone and give them an iPhone and they would do pretty well, if you did this with modern machine learning systems you will get an extremely high error rate. Or say you train an model on how to use a sword, I'm not convinced it would know how to use and ax or pair of crutches as a weapon.
Maybe it will turn out to simply be enough artificial neurons and everything works. But I don't believe that.
pyb · 12h ago
"... Since I am new to the research community, I made an effort"
This means they've probably submitted a paper too.
epolanski · 8h ago
It states it's a research, not a product company.
diggan · 8h ago
To be fair, OpenAI is also a "research lab" rather than "product company" and they still sell products for $200/month, not sure the distinction matters in practice much today as long as the entity is incorporated somehow.
pyb · 7h ago
That's what I said
saejox · 9h ago
What Carmack is doing is right. More people need to get away from training their models just with words. AI need the physicality.
NL807 · 5h ago
>AI need the physicality.
which i found interesting, because i remember Carmack saying simulated environments are way forward and physical environments are too impractical for developing AI
SeanaldMcDnld · 4h ago
Yeah in that way this demo seemed gimmicky like he acknowledged. He said in the past he would almost count people out if they weren’t training RL in a virtual environment. I agree, still happy he’s staying on the path of online continual learning though
programd · 3h ago
Nvidia seems to think the same thing. Here's Jim Fan talking about a "physical Turing test" and how embodied AI is the way forward.
He also talks needing large amounts of compute to run the virtual environments where you'll be training embodied AI. Very much worth watching.
steveBK123 · 6h ago
Another thought experiment - if OpenAI AGI was right around the corner, why are they wasting time/money/energy buying a product-less vanity hardware startup run by Ive?
Why not tackle robotics if anything.
Or really just be the best AGI and everyone will be knocking on your door to license it in their hardware/software stacks, you will print infinite money.
mindwok · 4h ago
AGI is not enough. Seriously, imagine if they had an AGI in their ChatGPT interface. It’s not enough to do anything truly meaningful. It’s like a genius in the woods somewhere. For AGI to have an impact it needs to be everywhere.
joshstrange · 22m ago
Once AGI is accomplished I can’t imagine what else it would do but bootstrap itself up which, depending on compute, could scale quite far. OpenAI would only need to feed it compute for the most part.
I don’t think AGI is close, but once it happens it’s hard to imagine it not “escaping” (whenever we want to define that as).
Jensson · 2h ago
> Seriously, imagine if they had an AGI in their ChatGPT interface. It’s not enough to do anything truly meaningful
If they had that people would make agents with it and then it can do tons of truly meaningful things.
People try to make agents with the current one but its really difficult since its not AGI.
steveBK123 · 4h ago
Robotics to navigate the physical world seems more impactful than some pin/glasses product to provide a passive audio/visual interface to the chatbot doesn't seem so earth shattering either though.
What would you do with a 10x or 100x smarter Siri/Alexa?
I still don't see my life changing.
Give me a robot that can legitimately do household errands like the dishes, laundry, etc.. now we are talking.
j_timberlake · 4h ago
This line of thought doesn't work, because any company approaching AGI might be actively trying to hide that information from regulators and the military. Being the 1st AGI company is actually pretty risky.
steveBK123 · 4h ago
VCs are far too conditioned as hype men to hide the ball like that.
After generations of boastful over-promising, do you really believe THIS time they are underpromising?
soared · 6h ago
Or have your AGI design products
steveBK123 · 6h ago
All the more reason not to acquihire Ive for $6.5B, if true
tiahura · 5h ago
Does AGI necessarily mean super-genius? Was KITT AGI? I'm not sure he could design products?
trendoid · 1h ago
No the term for that is ASI...artifical super intelligence. People in AI community have different timelines for that than AGI.
steveBK123 · 4h ago
Is VC really funding a trillion dollars of GPU purchases to replace labor that could instead be bid out to developing world mechanical turks for $1/hr?
soci · 4h ago
> Fundamentally, I believe in the importance of learning from a stream of interactive experience, as humans and animals do, which is quite different from the throw-everything-in-a-blender approach of pretraining an LLM. The blender approach can still be world-changingly valuable, but there are plenty of people advancing the state of the art there.
It's a shame that pretrained approach leads to such good enough result. The learning-from-experience, or what should be the "right" approach, will stagnate. I might be wrong, but it seems that aside from Carmack and a small team, "the world" is just not looking/investing on that side of the AI anymore.
However, I find it funny that Carmack is now researching for such approach. At the end of the day, he was the one who invented Portals, an algorithm to circumvent the need to reproduce the whole 3D world and therefore making 3D games computationally possible.
As a side note, I wonder what models are to come once we see the latest state of the art AI Video training technologies, in synch with the joystick movements from a real player. Maybe the results are so astonishing that even Carmack changes his mind on the subject.
EDIT::grammar & typos
tshaddox · 3h ago
> It's a shame that pretrained approach leads to such good enough result. The learning-from-experience, or what should be the "right" approach, will stagnate.
We’ll see. I’m skeptical that you’ll ever get novel theories like special and general relativity out of LLMs. For stuff like that I suspect you need the interactive learning approach, and perhaps more importantly, the ability to reject the current best theories and invent a replacement.
vlovich123 · 3h ago
I’m not necessarily convinced despite my human bias that it’s a superior mechanism. Humans work the way they do and learn the way they do in no small part because of biological limitations and a physical reality. It’s not clear that a virtual entity needs to face the same limitations, although clearly learning from feedback that’s not available to an AI is important. It is true though that humans are more energy efficient learners, but letting the AI experiment with the real world and get feedback that’s way may be the only missing piece rather than a problem with the “blender” approach.
anthonypasq · 4h ago
i think you're overstating this. Yann LeCun (chief scientist at Meta) is firmly in this camp, and i think most companies trying to bring AI into the real world via some sort of robotics technology are thinking about and testing this approach.
soci · 3h ago
Thank you. You are right, most likely the ones working in the field haven't switched.
But the truth is that big bucks are in pretrained technologies. As Carmack himself said, "there are plenty of people advancing the state of the art there".
koolala · 4h ago
Humans had 500 million * 8670 hours of Pre-Training.
I don't get why Carmack would say things should be learned in hours or upper bounds it to human lifetime.
flipnotyk · 3h ago
I think there's a difference between "should be learned" and "should be able to be learned" here.
xnx · 3h ago
I'm surprised there's as much interest in looking at the structure/behavior of the biological brain, and less interest in considering the behavior of our vision system. Our brains are not CPUs, and our eyes are definitely not a grid of pixels with a fixed framerate.
dusted · 12h ago
anywhere we can watch the presentation ? the slides alone are great, but if he's saying stuff alongside, I'd be interested in that too :)
mkoubaa · 5h ago
> It is worth trying out one of the many web based reaction time testers – you will find that you average over 160 milliseconds.
TIL JC has elite reflexes
2OEH8eoCRo0 · 5h ago
And nerves of steel
vasco · 12h ago
Bro went his whole career and managed to somehow create a gig for himself where he gets the AI money while playing Atari. It's hard to increase the respect for someone who you already maxed out on but there we go. Carmack is a cool guy.
willvarfar · 11h ago
Although Carmack is the quintessential not-a-brogrammer.
dusted · 11h ago
I've never actually seen a brogrammer though, I've seen people who program only because they get money for it, I thought for a while those where it, but I'm not sure if I think they qualify either.
tomaytotomato · 10h ago
From watching on the wall I've seen brogrammer used in various contexts (this is not an exhaustive list):
- Someone who is a programmer but follows a hypermasculine cliche and makes sure everyone knows about it.
- An insult used by other developers for someone who is more physically fit or interested in their health than themselves.
- An insult used by engineers or other people who are not happy with the over representation of men in the industry. So everyone is lumped in the category.
- Someone who is obsessed with the technology and trying to grind their skills on it to an excessive level.
brotein · 4h ago
Brogrammer here. I recognize that I spend 12 hours a day writing code (and loving it) as a fun thing but also a danger if I do it all sitting down. I stay busy and incorporate workouts into my day.
I don’t take it as a pejorative, it’s an acknowledgement of my efforts to be even considered in this category. For those wondering I have a family, and have healthy activities otherwise. No cool diets or bioscience, just code, physical activity and coffee/water.
This isn’t a lifestyle I’m saying everyone should do, only that people should do what makes them happiest and most fulfilled for their set of goals.
diggan · 8h ago
> someone who is more physically fit or interested in their health than themselves
Isn't "interested in their health" a signal that they are interested in themselves, rather than the opposite?
oersted · 7h ago
Disambigation: I believe "themselves" refers to the one insulting, not the one interested in their health.
It tripped me up too, to be fair.
tomaytotomato · 5h ago
Apologies, grammar is hard.
mi_lk · 9h ago
> Someone who is obsessed with the technology and trying to grind their skills on it to an excessive level
sounds like a person who respects their own profession though
shermantanktop · 3h ago
I always think of “bro” as being a hyper version of “dude.” It’s generically applied to any random person, but it’s also exclusively male. So using it implies “this ingroup is assumed to be 100% male.”
On the other hand, I’ve seen and heard “dude” and “guy” used by and applied to women by other women. Not common but it happens. But I’ve never heard “bro” used that way.
CPLX · 7h ago
A. A male programmer who uses the word "bro" unironically in conversation.
B. A person who is physically and culturally indistinguishable from A
rfrey · 5h ago
What physical or cultural characteristics would make a person "indistinguishable from A"?
nindalf · 11h ago
It means “programmer I don’t like”. Very versatile insult and vague enough that it’s impossible to defend against.
kid64 · 11h ago
No, I'm pretty sure it's just a programmer that understands the world in terms of bros.
dusted · 9h ago
"in the terms of bros" what does that even mean? I think bro is a term that's used pretty widely, for different things, in different cultures and contexts, I call my brother bro.. I've heard people call their friends bro.. I've heard someone tell a police officer "don't tase me, bro"..
floren · 3h ago
My co-worker's 7 year old daughter calls him "bruh"
lostmsu · 2h ago
I call my 4yo daughter bro
vasco · 7h ago
Expression of endearment in this case.
Flamentono2 · 6h ago
I find it interesting that he dismisses LLMs.
I would argue that if he wants to do AGI through RL, a LLM could be a perfect teacher or oracle.
After all i'm not walking around as a human and not having guidance. It should/could make RL a lot faster leveraging this.
My logical part / RL part does need the 'database'/fact part and my facts are trying to be as logical as possible but its just not.
akomtu · 3h ago
IMO, he's right. LLMs can't be AI because they don't create a model of observations to predict things, they just imitate observations based on their likeness to each other. When you play Quake, you create a simple model of the game physics and use that fast model to navigate the game. Your equivalent of LLM has a role too: it's a fuzzy detector of things you encounter in the game, sounds, images and symbols, but once detected, those things are fed into the fast and rigid physics model.
Quite exciting. Without diminishing the amazing value of LLMs, I don't think that path goes all the way to AGI. No idea if Carmack has the answer, but some good things will come out of that small research group, for sure.
petters · 12h ago
Isn't that what Deepmind did 12 years ago?
hombre_fatal · 11h ago
He points that out in his notes and says DeepMind needed specialized training and/or 200M frames of training just to kinda play one game.
tsunamifury · 11h ago
What deepmind accomplished with suicidal Mario was so much more than you probably ever will know from outside the company.
mi_lk · 9h ago
Do tell if you can. Were you there?
willvarfar · 11h ago
Playing Atari games makes it easy to benchmark and compare and contrast his future research with Deepmind and more recent efforts.
moralestapia · 11h ago
IIRC Deepmind (and OpenAI and ...) have done this on software-only setups (emulators, TAS, etc); while this one has live input and actuators in the loop, so, kind of the same thing but operating in the physical realm.
I do agree that it is not particularly groundbreaking, but it's a nice "hey, here's our first update".
No comments yet
lostmsu · 2h ago
I'm with OpenAI folks on this one: Atari just won't cut it for AGI. My layman intuition is that RL works well when rewards give good signal all the time. Until it does RL is basically random search. That's where massive data diversity like we have in text comes in handy.
In a game there might be a level with a door and a key, and because there's no reward for getting the key closer to the door, bridging this gap requires random search in a massive state space. But in the vast sea of scenarios that you can find in Common Crawl there's probably one, where you are 1 step from the key, and the key is 1 step from the door, so you get the reward signal from it without having to search an enormous state space.
You might say "but you have to search through the giant Common Crawl". Well yes, but while doing so you will get reward signal not just for the key and door problem, but for nearly every problem in the world.
The point is: pretraining teaches models to extract signal that can be used to explore solutions to hard search problems, and if you don't do that you are wasting your time enumerating giant state spaces.
lostmsu · 2h ago
You can actually easily test and overcome this by training a model simultaneously on a massive of text and Atari while carefully balancing learning rates between the two.
abc-1 · 11h ago
I was really hoping to see something cool. Such a great team of smart people, but this is just John reverting to what he’s comfortable with and going off on some nonsense tangent. Hardware and video games. I doubt this is going to yield anything interesting at all.
rfrey · 5h ago
From chess to Go to Atari games, AI research has always been about games. You are unintentionally positioning yourself as more sophisticated than a huge number of AI luminaries, including Nobel and Turing winners.
abc-1 · 3h ago
Feel free to actually respond to what I’m discussing. There is little to no value in doing real time video games using a physical input device at this stage in AI.
shermantanktop · 3h ago
Maybe, but many advances in science have come from individuals moving backward from the popular cutting edge and branching off in a new direction.
johnb231 · 11h ago
Games are heavily used in RL research.
abc-1 · 10h ago
I understand that. Doing games in real time is just a performance problem that can be solved with more compute or inane optimizations. It’s not interesting research.
johnb231 · 10h ago
I think you are trivializing the field of RL research. Games are not a solved problem. Doing that efficiently in real-time is even more difficult and is highly relevant to real world applications.
abc-1 · 10h ago
Right, we have not even solved games in the non-real time environment, so why bother adding additional constraints like “on real hardware in real time”. This is exactly like Tesla trying to switch from LIDAR to cameras before self driving is even solved. It’s avoiding the real harder challenge and going off on inane tangents. John is essentially bike shedding.
In this case, John is going off on this inane tangent because of his prior experience with hardware and video games instead of challenging himself to solve the actual hard and open problems.
I’m going to predict how this plays out for the inevitable screenshot in one to two years. John picks some existing RL algo and optimizes it to run in real time on real hardware. While he’s doing this the field moves on to better and new algorithms and architectures. John finally achieves his goal and posts a vid of some (now ancient) RL algo playing some Atari game in real time. Everyone says “neat” and moves on. John gets to feel validated yet all his work is completely useless.
johnb231 · 9h ago
False dichotomy. It's not "avoiding the real harder challenge". It's solving a different problem and it is extremely relevant to real world applications. These are actual hard and open problems to be solved.
criddell · 5h ago
Here's a heuristic that somebody gave me a while ago: using the word "just" in the way you did is a signal that you don't understand the topic.
John's document covers why he's doing what he's doing:
> Fundamentally, I believe in the importance of learning from a stream of interactive experience, as humans and animals do, which is quite different from the throw-everything-in-a-blender approach of pretraining an LLM. The blender approach can still be world-changingly valuable, but there are plenty of people advancing the state of the art there.
He thinks interacting with the real world and learning as you go isn't getting enough attention and might take us farther than the LLM approach. So he's applying these ideas to a subject that he's an expert in. You don't seem to find this approach interesting but John does (and I do too, for the record).
Everybody dismissing him might be right. Those keeping score know that Carmack's batting average isn't one thousand. But those people also know Carmack has the resources to work on pretty much whatever he wants to work on. I'm happy he's still working hard at something and sharing his work.
dylan604 · 4h ago
for systems that learn in real-time, is there a way for the humans to know/understand how/why the system came to the conclusion it has? there are examples of where humans ran experiments that came to a conclusion for the wrong reasons. if an AI system thinks it knows the answer for the wrong reason, wouldn't that then poison its reasoning later as well? can an AI system learn its reasoning is wrong and then update it when provided better evidence? that seems to be something a vast majority of humans cannot do.
abc-1 · 3h ago
Oh darn I used the word just now everything I said is invalidated! That’s too bad!
FWIW, I’m aware of that heuristic too which is why I intentionally use the word “just” as a meta heuristic to filter a certain type of person.
riehwvfbk · 5h ago
And this is different from the argument that's being pooh-poed and downvoted how? You are effectively saying "Carmack is smart and is working on cool stuff that'll most likely be useless" in different words.
criddell · 4h ago
Not sure what gave you that impression. It's definitely not what I was trying to say.
brador · 11h ago
I feel top level AI creation is beyond his skill set.
He’s a AAA software engineer but the prerequisites to build out cutting edge AI require deep formal math that is beyond his education and years at this point.
Nothing to stop him playing around with AI models though.
kriro · 10h ago
I think you overestimate the level of math required in AI and at the same time I think you underestimate the math skills of John. AI runs on GPUs, Quake 2 engine was one of the first to optimized for GPUs (OpenGL).
I'm pretty excited to see him in this domain. I think he'll focus on some DeepSeek style improvements.
horsellama · 10h ago
this.
Having JC focusing on, say, writing a performant OSS CUDA replacement could be bigger than any of the last 20 announcements from openai/goggle/deepmind/etc
lyu07282 · 9h ago
This feels like an understatement. At the time, young me had the impression Carmack came first, then the industry created 3dfx/OpenGL to run his games better. I still have nothing but respect for his skills decades later.
threeseed · 7h ago
John Carmack:
So I asked Ilya, their chief scientist, for a reading list. This is my path, my way of doing things: give me a stack of all the stuff I need to know to actually be relevant in this space.
And he gave me a list of like 40 research papers and said, 'If you really learn all of these, you'll know 90% of what matters today! And I did. I plowed through all those things and it all started sorting out in my head.
abraxas · 3h ago
I'd love to have a copy of that list. Just to see how much I've yet to absorb.
foldr · 7h ago
What this misses is that research is a competitive endeavor. To succeed as a researcher you don’t just need to know the bare minimum required to do research in your field. You need to be able to do it better than most of the people you’re competing against. I know that HN as a collective has near-unlimited faith in Carmack’s abilities (and he is no doubt Very Smart). But he’s competing with other Very Smart people who have decades more experience of AI research.
To put it another way, the idea that John Carmack is going to do groundbreaking research in AI is roughly as plausible as the idea that Yann LeCun is going to make a successful AAA video game. Stranger things have happened, but I won’t be holding my breath.
RetroTechie · 6h ago
You're forgetting that a whole string of breakthroughs are all fairly recent (like in the last decade). Everyone, including the pro's, is treading new ground.
In that context anyone can make progress in the field, as long as they understand what they're dealing with.
Better regard mr. Carmack as an X factor. Maybe the experts will leave him in the dust. Or maybe he'll come up with something that none of the experts cared to look into.
foldr · 5h ago
Lots of scientific fields have seen breakthroughs in the past decade. Doesn’t mean that any random smart person can jump in and start doing groundbreaking research.
Jensson · 2h ago
But a random smart person will jump in and make groundbreaking research.
secondcoming · 6h ago
Why does it need to be competitive? Maybe the guy has enough money to let him do whatever he wants regardless of the outcome and he chose AI because it's interesting to him.
foldr · 5h ago
In research you have to succeed before your competitors. It’s not research if it’s already been done.
No comments yet
Cthulhu_ · 9h ago
What do you mean "beyond his skill set"? He effectively invented 3D gaming, which led to major leaps and investments into graphics cards which are now used for cryptocurrency and AI. He also did significant contributions into VR.
He's probably one of the most qualified people around.
jimbohn · 8h ago
The math around machine learning is very manageable, and a lot of research in that area is throwing heuristics at a wall to see what sticks
johnb231 · 11h ago
The formal math takes a few months to learn. He is more than smart enough to figure that out.
novosel · 11h ago
There is no deep formal math in AI. It is a game of numbers.
All deep formal math is a boundary to a thing.
akomtu · 3h ago
AI creation is more like Alchemy than science, and breakthroughs come not from math background, but from intuition and a bit of math skills. Transformers behind the chatbots isn't a rocket science and were discovered almost by accident. The next breakthrough will come a similar way. I'd frankly bet on someone like Carmack than on some theoretical researcher who is churning out papers.
roflcopter69 · 7h ago
Honestly, having gone through the slides, it's a bit painful to see Carmack "rediscover" stuff I've learned in a reinforcement learning lecture like ten years ago.
But don't get me wrong! Since this is a long-term research endeavor of his, I believe really starting from the basics is good for him and will empower him to bring something new to the table eventually.
I'm surprised though that he "only" came so far as of now. Maybe my slight idolization of Carmack made me kinda of blind to the fact that this kind of research is a mean beast after all and there is a reason that huuuuge research labs dump countless of man-decades into this kind of stuff with no guaranteed breakthroughs.
I'm nowhere as good at my craft as someone who works for openai, which the author of that tweet seems to be, but if even I can see this, then it's bad, isn't it?
No comments yet
xiphias2 · 12h ago
A lot of the problems John mentioned (camera jpeg, latency, real time decisions) have been worked on by comma.ai for many years. He could have just used their stack and build on it the general learning parts that comma is not focusing on.
WatchDog · 9h ago
Carmack himself has done a lot of work on end to end latency, during his time at oculus.
Cthulhu_ · 9h ago
This is Carmack, who builds new things. He built one of the first 3D game engines based on the hard math.
prosunpraiser · 11h ago
Reuse is not always necessary - sometimes things are just done for fun and exploration, not for appeasing thirsty VCs and grabbing that market share.
blitzar · 10h ago
Reinventing the exact same thing and shouting from the rooftops about it is exactly how you appease thirsty VCs and grab that market share.
CPLX · 7h ago
Might I interest you in my new startup, that is a bus, but with technology?
Flemlo · 11h ago
And plenty of other people.
It's still a lot better to really learn and discover it yourself to really get it.
Also it's hard to determine how much time someone spent on particular topic.
aatd86 · 11h ago
My own little insights and ramblings as an uninitiated quack (just spent the night asking Claude to explain machine learning to me):
seems that we are learning in layers, one of the first layers being 2D neural net (images) augmented by other sensory data to create a 3D if not 4D model (neural net).
HRTFs for sound increases the spatial data we get from images.
With depth coming from sound and light and learnt movements(touch) we seem to develop a notion of space and time. (multimodality?)
Seems that we can take low dimensional inputs and correlate them to form higher dimensional structures.
Of course, physically it comes from noticing the dampening of visual data (in focus for example) and memorized audio data (sound frequency and amplitude, early reflections, doppler effect etc).
That should be emergent from training.
Those data sources can be inperfectly correlated. That's why we count during a lightning storm to evaluate distance. It's low dimensional.
In a sense, it's a measure of required effort perhaps (distance to somewhere).
What's funny is that it seems to go the other way from traditional training where we move from higher dimensional tensor spaces to lower ones. At least in a first step.
Flamentono2 · 6h ago
Its hard to follow what you try to commounicate at least the last half.
Nonetheless, yes we do know certain brain structures like your image net analogy but the way you describe it, sounds a little bit of.
Our virtual cortex is not 'just a layer' its a component i would say and its optimized of detecting things.
Other components act differently with different structures.
epr · 5h ago
A bit confusing for sure, but I think (not sure) I get what they're saying. Training a nn (for visual tasks at least) consists of training a model with much more dimensions (params) than the input space (eg: controller inputs + atari pixels). This contrasts with a lot of what humans do, which is take higher dimensional information (tons of data per second combining visual, audio, touch/vibration, etc) and synthesizing much lower dimensional models / heuristics / rules of thumb, like the example they give of the 5 second per mile rule for thunder.
One question I would have about the research direction is the emphasis on realtime. If I understand correctly he's doing online learning in realtime. Obviously makes for a cool demo and pulls on his optimisation background, and no doubt some great innovations will be required to make this work. But I guess the bitter lesson and recent history also tell us that some solutions may only emerge at compute levels beyond what is currently possible for realtime inference let alone learning. And the only example we have of entities solving Atari games is the human brain, of which we don't have a clear understanding of the compute capacity. In which case, why wouldn't it be better to focus purely on learning efficiency and relax the realtime requirement for now?
That's a genuine question by the way, definitely not an expert here and I'm sure there's a bunch of value to working within these constraints. I mean, jumping spiders solve reasonably complex problems with 100k neurons, so who knows.
Obviously both Carmack and the rest of the world has changed since then, but it seems to me his main strength has always been in doing more with less (early id/Oculus, AA). When he's working in bigger orgs and/or with more established tech his output seems to suffer, at least in my view (possibly in his as well since he quit both Bethesda-id and Meta).
I don't know Carmack and can't claim to be anywhere close to his level, but as someone also mainly interested in realtime stuff I can imagine he also feels a slight disdain for the throw-more-compute-at-it approach of the current AI boom. I'm certainly glad he's not running around asking for investor money to train an LLM.
Best case scenario he teams up with some people who complement his skillset (akin to the game designers and artists at id back in the day) and comes up with a way to help bring some of the cutting edge to the masses, like with 3D graphics.
What Carmack did was basically get a 3d game running on existing COMMODITY hardware. The 386 chip that most people used for their excel spreadsheets did not do floating point operations well, so Carmack figured out how to do everything using integers.
May 1992 -> Wolfenstein 3d releases December 1993 -> Doom releases December 1994 -> Sony Playstation launches in Japan June 1996 -> Quake releases
So Wolfenstein and Doom were actually not really 3d games, but rather 2.5 games (you can't have rooms below other rooms). The first 3d game here is actually Quake which also eventually also got hardware acceleration support.
Carmack was the master of doing the seeminly impossible on super constrained hardware on virtually impossible timelines. If DOOM released in 1994 or 1995, would we still remember it in the same way?
Maybe. One aspect of Wolfenstein and Doom's popularity is that it was years ahead of everyone else technically on PC hardware. The other aspect is that they were genre defining titles that set the standards for gameplay design. I think Doom Deathmatch would have caught on in 1995, as there really were very few (just Command and Conquer?) standout PC network multiplayer games released between 1993 and 1995.
The first 3d console games started to come out that year, like Rayman. Star Wars Dark Forces with its own custom 3d engine also came out. Of course Dark Forces was, however, an overt clone of DOOM.
It's a bit ironic, but I think the gameplay innovation of DOOM tends to hold up more than the actual technical innovation. Things like BSP for level partitioning have slowly been phased out of game engines, we have ample floating point compute power and hardware acceleration ow, but even developers of the more recent DOOM games have started to realize that they should return to the original formula of "blast zombies in the face at high speed, and keep plot as window dressing"
I think so, because the thing about DOOM is, it was an insanely good game. Yes, it pioneered fullscreen real-time perspective rendering on commodity hardware, instantly realigning the direction of much of the game industry, yadda yadda yadda, but at the end of the day it was a good-enough game for people to remember and respect even without considering the tech.
Minecraft would be a similar example. Minecraft looked like total ass, and games with similar rendering technology could have been (and were) made years earlier, but Minecraft was also good. And that was enough.
"A reality check for people that think full embodied AGI is right around the corner is to ask your dancing humanoid robot to pick up a joystick and learn how to play an obscure video game."
Each of those 100 can hire teams or colleagues to make their domain better, so there’s always human expertise keeping the model updated.
200 million to have dedicated top experts on hand is reasonable.
From what I can tell most in AI are currently hoping LLMs reach that point quick just because the hype is not helping AI at all.
You can call this hype, maybe it is all hype until LLMs can work on 10M LOC codebases, but recognize that LLMs are a shift that is totally incomparable to any previous AI advancement.
I'd definitely like to improve my skills, but to be realistic, most of the programmers are not top-notch.
It will help a single human do more in a white collar world.
https://arxiv.org/abs/2303.10130
Sounds like AI robbed you of an opportunity to spend some time with your Dad, to me
It takes five minutes to program the thermostat, then you can have a beer on the patio if that's your speed and catch up for a bit
Life is little moments, not always the big commitments like taking a day to go fishing
That's the point of automating all of ourselves out of work, right? So we have more time to enjoy spending time with the people we love?
So isn't it kind of sad if we wind up automating those moments out of our lives instead?
It's not about specialized vs generalized models - it's about how models are trained. The chess engine that beat Kasparov is a specialized model (it only plays chess), yet it's the bitter lesson's example for the smarter way to do AI.
Chess engines are better at chess than LLMs. It's not close. Perhaps eventually a superintelligence will surpass the engines, but that's far from assured.
Specialized AI are hardly obsolete and may never be. This hypothetical superintelligence may even decide not to waste resources trying to surpass the chess AI and instead use it as a tool.
I hate Adobe, I don’t like to give them credit for anything. But their audio enhance tool is actual sorcery. Every competitor isn’t even close. You can take garbage zoom audio and make it sound like it was borderline recorded in a treated room/studio. I’ve been in production for almost 15 years and it would take me half a day or more of tweaking a voice track with multiple tools that cost me hundreds of dollars to get it 50% as good as what they accomplish in a minute with the click of a button.
Because we evolved to get where we are, humans have all sorts of messy behaviours that aren't really compatible with a utopian society. Theft, violence, crime, greed - it's all completely unnecessary and yet most of us can't bring ourselves to solve these problems. And plenty are happy to live apathetically while billionaires become trillionaires...for what exactly? There's a whole industry of hyper-luxury goods now, because they make so much money even regular luxury is too cheap.
If we can produce AGI that exceeds the capabilities of our species, then my hope is that rather than the typical outcome of "they kill us all", that they will simply keep us in line. They will babysit us. They will force us all to get along, to ensure that we treat each other fairly.
As a parent teaches children to share by forcing them to break the cookie in half, perhaps AI will do the same for us.
Funnily enough, I still think some of the most interesting semi-recent writing on utopia was done ~15 years ago by... Eliezer Yudkowsky. You might be interested in the article on "Amputation of Destiny."
Link: https://www.lesswrong.com/posts/K4aGvLnHvYgX9pZHS/the-fun-th...
It would seem our own generalized intelligence is an emergent property of many, _many_ specialized processes
I wonder if AI is the same
You can say that about other animals, but about humans it is not so sure. No animal can be taught as general set of skills as a human can, they might have some better specialized skills but clearly there is something special that makes humans so much more versatile.
So it seems there was this simple little thing humans got that makes them general, while for example our very close relatives the monkeys are not.
Science is full of theories that are correct per our current knowledge and then subsequently disproven when research/methods/etc improves.
Humans aren't special, we are made from blood & bone, not magic. We will eventually build AGI if we keep at it. However unlike VCs with no real skills except having a lot of money™, I couldn't say whether this is gonna happen in 2 years or 2000.
I made a google form question for collecting AGI definitions cause I don't see anyone else doing it and I find it infinitely frustrating the range of definitions for this concept:
https://docs.google.com/forms/d/e/1FAIpQLScDF5_CMSjHZDDexHkc...
My concern is that people never get focused enough to care to define it - seems like the most likely case.
It's an ideal that some people believe in, and we're perpetually marching towards it
Can we just use Morris et al and move on with our lives?
Position: Levels of AGI for Operationalizing Progress on the Path to AGI: https://arxiv.org/html/2311.02462v4
There are generational policy and societal shifts that need to be addressed somewhere around true Competent AGI (50% of knowledge work tasks automatable). Just like climate change, we need a shared lexicon to refer to this continuum. You can argue for different values of X but the crucial point is if X% of knowledge work is automated within a decade, then there are obvious risks we need to think about.
So much of the discourse is stuck at “we will never get to X=99” when we could agree to disagree on that and move on to considering the x=25 case. Or predict our timelines for X and then actually be held accountable for our falsifiable predictions, instead of the current vide based discussions.
There is no point collecting definitions for AGI, it was not conceived as a description for something novel or provably existent. It is "Happy Meal marketing" but aimed for adults.
The term AGI is obviously used very loosely with little agreement to it's precise definition, but I think a lot of people take it to mean not only generality, but specifically human-level generality, and human-level ability to learn from experience and solve problems.
A large part of the problem with AGI being poorly defined is that intelligence itself is poorly defined. Even if we choose to define AGI as meaning human-level intelligence, what does THAT mean? I think there is a simple reductionist definition of intelligence (as the word is used to refer to human/animal intelligence), but ultimately the meaning of words are derived from their usage, and the word "intelligence" is used in 100 different ways ...
My masters thesis advisor Ben Goertzel popularized the term and has been hosting the AGI conference since 2008:
https://agi-conference.org/
https://goertzel.org/agiri06/%5B1%5D%20Introduction_Nov15_PW...
I had lunch with Yoshua Bengio at AGI 2014 and it was most of the conversation that day
Games generally are solvable for AI because they have feedback loops and a clear success or failure criteria. If the "picking up a Joystick" part is the limiting factor, sure. But why would we want robots to use an interface (especially a modern controller) heavily optimized for human hands; that seems like the definition of a horseless carriage.
I'm sure if you compared a monkey and a dolphins performance using a joystick you'd get results that aren't really correlated with their intelligence. I would guess that if you gave robots an R2D2 like port to jack into and play a game, that problem could be solved relatively quickly.
They also claimed it "learned" to play by playing itself only however it was clear that most of the advanced techniques were borrowed from existing AI and by observing humans.
No surprise they gave up on that project completely and I doubt they'll ever engage in anything like that again.
Money better spent on different marketing platforms.
Saying you've solved Dota after stripping out nearly all of its complexity is like saying you've solved Chess, but on a version where the back row is all Bishops.
Look at how long Theranos went on! Miraculous product. Attractive young founder with all the right pedigree, credentials, and contacts, dressed in black trurtlenecks. Hell, she even talked like Steve Jobs! Investors never had a chance.
That is what investors see. You seem to treat this as a purity contest where you define purity
No comments yet
As an ex dota player, I don't think this is that far off from having full on, all heroes dota. Certainly not as far of as you are making it sound.
And dota is one of the most complex games, I expect for example that an AI would instantly solve CS since aim is such a large part of the game.
Only the first time, later when it played better players it always lost. Players learned the faults of the AI after some time in game and the AI had very bad late game so they always won later.
It is certainly possible, but i won't be impressed by anything "playing CS" that isn't running a vision model on a display and moving a mouse, because that is the game. The game is not abstractly reacting to enemy positions and relocating the cursor, it's looking at a screen, seeing where the baddy is and then using this interface (the mouse) to get the cursor there as quickly as possible.
It would be like letting an AI plot its position on the field and what action its taking during a football match and then saying "Look, The AI would have scored dozens of times in this simulation, it is the greatest soccer player in the world!" No, sorry, the game actually requires you to locomote, abstractly describing your position may be fun but it's not the game
But then again, that is precisely the point. A chess bot also has access to gigabytes of perfect working memory. I don't see people complaining about that. It's perfectly valid to judge the best an AI can do vs the best a human can do. It's not really fair to take away exactly what a computer is good at from an AI and then say: "Look but the AI is now worse". Else you would also have to do it the other way around. How well could a human play dota if it only had access to the bot API. I don't think they would do well at all.
As we're learning with LLMs, the dataset is what matters - and what's awesome is that you can see that in us, as well! I've read that our evolution is comparatively slow to the rate of knowledge accumulation in the information age - and that what this means is that you can essentially take a caveman, raise them in our modern environment and they'll be just as intelligent as the average human today.
But the core of our intelligence is logic/problem solving. We just have to solve higher order problems today, like figuring out how to make that chart in excel do the thing you want, but in days past it was figuring out how to keep the fire lit when it's raining. When you look at it, we've possessed the very core of that problem solving ability for quite a while now. I think that is the key to why we are human, and our close ancestors monkeys are...still just monkeys.
It's that problem solving ability that we need to figure out how to produce within ML models, then we'll be cooking with gas!
Elon's response to this is that if we want these androids to replace human jobs then the lowest friction alternative is for the android to be able to do anything a human can do in a human amount of space. A specialized machine is faster and more efficient, but comes with engineering and integration costs that create a barrier to entry. Elon learned this lesson the hard way when he was building out the gigafactories and ended up having to hire a lot of people to do the work while they sorted out the issues with the robots. To someone like Elon a payroll is an ever growing parasite on a companies bottom line, far better if the entire thing is automated.
AI clearly isn't at human level and it's OK to admit it.
Hundreds of millions of years of trial-and-error biological pre-training where survival/propagation is the reward function
It just isn't needed. Just like you can find let's say kangaroos in the latent space of an image generator, so we learn abstract concepts and principles of how things work as a bonus of learning to process the senses.
Maybe a way to AGI could be figuring out how to combine a video generator with a LLM or something similar in a way that allows it to understand things intuitively, instead of doing just lots and lots of some statistical bullsit.
We do have that, ever felt fear of heights? That isn't learned, we are born with it. Same with fear of small moving objects like spiders or snakes.
Such things are learned/stored very different from memories, but its certainly there and we can see animals also have those. Like cats gets very scared of objects that are long and appear suddenly, like a cucumber, since their genetic instincts thinks its a snake.
After having raised four dozen kittens that a couple of feral sisters gave birth to in my garage, I’m certain that is nonsense. It’s an internet meme that became urban legend.
I don’t think they have ever even reacted to a cucumber, and I have run many experiments because my childhood cat loved cucumbers (we’d have to guard the basket of cucumbers after harvest, otherwise she’d bite every single one of them… just once).
I don't think it's clear how much of a human brains function exists at birth though, I know it's theorised than even much of the sensory processing has to be learned.
Existing at birth is not the same thing as innate. Puberty is innate but it is not present at birth.
Neurons have finite (very low) speed of signal transfer, so just by measuring cognitive reaction time we can deduce upper bounds on how many _consecutive_ neuron connections are involved in reception, cognitive processing, and resulting reaction via muscles, even for very complex cognitive processes. And the number is just around 100 consecutive neurons involved one after another. So “the algorithm” could not be _that_ complex in the end (100x matmul+tanh?)
Granted, a lot of parallelism and feedback loops are involved, but overall it gives me (and many others) an impression that when the AGI algorithm is ever found, it’s “mini” version should be able to run on modest 2025 hardware in real time.
Biological neurons are way more complex than that. A single neuron has dentritic trees with subunits doing their own local computations. There are temporal dynamics in the firing sequences. There is so much more complexity in the biological networks. It's not comparable.
https://docs.google.com/presentation/d/1GmGe9ref1nxEX_ekDuJX...
https://docs.google.com/document/d/1-Fqc6R6FdngRlxe9gi49PRvU...
> I read through these slides and felt like I was transported back to 2018.
> Having been in this spot years ago, thinking about what John & team are thinking about, I can't help but feel like they will learn the same lesson I did the hard way.
> The lesson: on a fundamental level, solutions to these games are low-dimensional. No matter how hard you hit them with from-scratch training, tiny models will work about as well as big ones. Why? Because there's just not that many bits to learn.
> If there's not that many bits to learn, then researcher input becomes non-negligible.
> "I found a trick that makes score go up!" -- yeah, you just hard-coded 100+ bits of information; a winning solution is probably only like 1000 bits. You see progress, but it's not the AI's.
> In this simplified RL setting, you don't see anything close to general intelligence. The neural networks aren't even that important.
> You won't see _real_ learning until you absorb a ton of bits into the model. The only way I really know to do this is with generative modeling.
> A classic example: why is frame stacking just as good as RNNs? John mentioned this in his slides. Shouldn't a better, more general architecture work better?
> YES, it should! But it doesn't, because these environments don't heavily encourage real intelligence.
Systems that can learn to play Atari efficiently are exploiting the fact that the solutions to each game are simple to encode (compared to real world problems). Furthermore, you can nudge them towards those solutions using tricks that don't generalize to the real world.
“The lesson: on a fundamental level, solutions to these games are low-dimensional. No matter how hard you hit them with from-scratch training, tiny models will work about as well as big ones. Why? Because there's just not that many bits to learn.”
https://unrollnow.com/status/1925795730150527191
Graphics rendering and AI live on the same pyramid of technology. A pyramid with a lot of bricks with the initials "JC" carved into them, as it turns out.
Carmack is a genius no doubt. But genius is the result of intense focused practice above and beyond anyone else in a particular area. Trying to extend that to other domains has been the downfall of so many others like him.
No comments yet
However making a system that can beat an unknown game does require generalization. If not real a intelligence (whatever that means) but at the level of say "a wolf".
Whether it can arise from RL alone is not certain, but it's there somewhere.
And I say this while most certainly not being as knowledgeable as this openai insider. So it even I can see this, then it's kinda bad, isn't it?
Using real life robots is going to be a huge bottleneck for training hours no matter what they do.
https://github.com/Farama-Foundation/Arcade-Learning-Environ...
The goal is to develop algorithms that generalize to other tasks.
[1]https://github.com/cshenton/atari-leaderboard
I think we could eventually saturate Atari, but for now it looks like it's still a good source of problems that are just out of reach of current methods.
Continuous training is the key ingredient. Humans can use existing knowledge and apply it to new scenarios, and so can most AI. But AI cannot permanently remember the result of its actions in the real world, and so its body of knowledge cannot expand.
Take a toddler and an oven. The toddler has no concept of what an oven is other than maybe that it smells nice. The toddler will touch the oven, notice that it experiences pain (because the oven is hot) and learn that oven = danger. Place a current AI in a droid toddler body? It will never learn and keep touching the oven as soon as the information of "oven = danger" is out of the context window.
For some cases this inability to learn is actually desirable. You don't want anyone and everyone to be able to train ChatGPT unsupervised, otherwise you get 4chan flooding it with offensive crap like they did to Tay [1], but for AI that physically interacts with the meatspace, constant evaluation and learning is all but mandatory if it is to safely interact with its surroundings. "Dumb" robots run regular calibration cycles for their limbs to make sure they are still aligned to compensate for random deviations, and so will AI robots.
[1] https://en.wikipedia.org/wiki/Tay_(chatbot)
It is, at least if you wish to be in the meatspace, that's my point. Every day has 86400 seconds during which a human brain constantly adapts to and learns from external input - either directly as it's being awake or indirectly during nighttime cleanup processes.
On top of that, humans have built-in filters for training. Basically, we see some drunkard shouting about the Hollow Earth on the sidewalk... our brain knows that this is a drunkard and that Hollow Earth is absolutely crackpot material, so if it stores anything at all then the fact that there is a drunkard on that street and one might take another route next time, but the drunkard's rambling is forgotten maybe five minutes later.
AI, in contrast, needs to be hand-held by humans during training that annotate, "grade" or weigh information during the compilation of the training dataset, in order that the AI knows what is written in "Mein Kampf" so it can answer questions upon it, but that it also knows (or at least: won't openly regurgitate) that the solution to economic problems isn't to just deport Jews.
And huge context windows aren't the answer either. My wife says me, she would like to have a fruit cake for her next birthday. I'll probably remember that piece of information (or at the very least I'll write it down)... but an AI butler? I'd be really surprised if this is still in its context space in a year, and even if it is, I would not be surprised if it weren't able to recall that fact.
And the final thing is prompts... also not the answer. We've seen it just a few days ago with Grok - someone messed with the system prompt so it randomly interjected "white genocide" claims into completely unrelated conversation [1] despite hopefully being trained on a ... more civilised dataset, and to the contrary, we've also seen Grok reply to Twitter questions in a way that suggest that it is aware its training data is biased.
[1] https://www.reuters.com/business/musks-xai-updates-grok-chat...
That's not even remotely true. At least not in the sense that it is for context in transformer models. Or can you tell me all the visual and auditory inputs you experienced yesterday at the 45232nd second? You only learn permanently and effectively from particular stimulation coupled with surprise. That has a sample rate which is orders of magnitude lower. And it's exactly the kind of sampling that can be replicated with a run-of-the-mill persistent memory system for an LLM. I would wager that you could fit most people's core experiences and memories that they can randomly access at any moment into a 1000 page book - something that fits well into state of the art context windows. For deeper more detailed things you can always fall back to another system.
Just because I don't remember my experiences at second 45232 on May 22, doesn't mean that my brain was not actively adapting to my experiences at that moment. The brain does a lot more learning than just what is conscious. And then when I went to sleep the brain continued pruning and organizing my unconscious learning for the day.
Seeing if someone can go from token to freeform physical usefulness will be interesting. I'm of the belief that LLMs are too verbose and energy intensive to go from language regurgitation machines to moving in the real world according to free form prompting. It may be accomplishable with the vast amount of hype investment, but I think the energy requirements and latency will make an LLM-based approach economically infeasible.
This is just, not true. A single 2min conversation with emotional or intellectual resonance can significantly alter a human’s thought process for years. There are some topics where every time they come up directly or analogously I can recall something a teacher told me in high school that “stuck” with me for whatever reason. And it isn’t even a “core” experience, just something that instantly clicked for my brain and altered my problem solving. At the time, there’s no heuristic that could predict how or why that particular interaction should have that kind of staying power.
Not to mention, experiences that subtly alter thinking or behavior just by virtue of providing some baseline familiarity instead of blank slate problem solving or routine. Like how you subtly adjust how you interact with coworkers based on the culture of your current company over time vs the last without any “flash” of insight required.
Doesn't the article states that this is not true? AI cannot apply to B what it learned about A.
That's essentially what we're looking for when we talk about general intelligence, the capability to adapting what we know to what we know nothing about.
This is a continuous process.
His goal is to develop generic methods. So you could work with more complex games or the physical world for that, as that is what you want in the end. However, his insight is, you can even modify the Atari setting to test this, e.g. to work in realtime, and the added complexity by more complex games doesn't really give you any new additional insights at this point.
The approach NVIDIA are using (and other labs) clearly works. It's not going to be more than a year or two now before robotics is as solved as NLP and chatbots are today.
If it is substantially more sample efficient, or generalizable, than prior work then that would be exciting. But I'm not sure if it is?
If so, scaling up may be more of a distraction rather than helpful (besides wasting resources).
I hope he succeeds in whatever he's aiming for.
Maybe it will turn out to simply be enough artificial neurons and everything works. But I don't believe that.
which i found interesting, because i remember Carmack saying simulated environments are way forward and physical environments are too impractical for developing AI
https://www.youtube.com/watch?v=_2NijXqBESI
He also talks needing large amounts of compute to run the virtual environments where you'll be training embodied AI. Very much worth watching.
Why not tackle robotics if anything. Or really just be the best AGI and everyone will be knocking on your door to license it in their hardware/software stacks, you will print infinite money.
I don’t think AGI is close, but once it happens it’s hard to imagine it not “escaping” (whenever we want to define that as).
If they had that people would make agents with it and then it can do tons of truly meaningful things.
People try to make agents with the current one but its really difficult since its not AGI.
What would you do with a 10x or 100x smarter Siri/Alexa? I still don't see my life changing.
Give me a robot that can legitimately do household errands like the dishes, laundry, etc.. now we are talking.
After generations of boastful over-promising, do you really believe THIS time they are underpromising?
It's a shame that pretrained approach leads to such good enough result. The learning-from-experience, or what should be the "right" approach, will stagnate. I might be wrong, but it seems that aside from Carmack and a small team, "the world" is just not looking/investing on that side of the AI anymore.
However, I find it funny that Carmack is now researching for such approach. At the end of the day, he was the one who invented Portals, an algorithm to circumvent the need to reproduce the whole 3D world and therefore making 3D games computationally possible.
As a side note, I wonder what models are to come once we see the latest state of the art AI Video training technologies, in synch with the joystick movements from a real player. Maybe the results are so astonishing that even Carmack changes his mind on the subject.
EDIT::grammar & typos
We’ll see. I’m skeptical that you’ll ever get novel theories like special and general relativity out of LLMs. For stuff like that I suspect you need the interactive learning approach, and perhaps more importantly, the ability to reject the current best theories and invent a replacement.
I don't get why Carmack would say things should be learned in hours or upper bounds it to human lifetime.
TIL JC has elite reflexes
- Someone who is a programmer but follows a hypermasculine cliche and makes sure everyone knows about it.
- An insult used by other developers for someone who is more physically fit or interested in their health than themselves.
- An insult used by engineers or other people who are not happy with the over representation of men in the industry. So everyone is lumped in the category.
- Someone who is obsessed with the technology and trying to grind their skills on it to an excessive level.
I don’t take it as a pejorative, it’s an acknowledgement of my efforts to be even considered in this category. For those wondering I have a family, and have healthy activities otherwise. No cool diets or bioscience, just code, physical activity and coffee/water.
This isn’t a lifestyle I’m saying everyone should do, only that people should do what makes them happiest and most fulfilled for their set of goals.
Isn't "interested in their health" a signal that they are interested in themselves, rather than the opposite?
It tripped me up too, to be fair.
sounds like a person who respects their own profession though
On the other hand, I’ve seen and heard “dude” and “guy” used by and applied to women by other women. Not common but it happens. But I’ve never heard “bro” used that way.
B. A person who is physically and culturally indistinguishable from A
I would argue that if he wants to do AGI through RL, a LLM could be a perfect teacher or oracle.
After all i'm not walking around as a human and not having guidance. It should/could make RL a lot faster leveraging this.
My logical part / RL part does need the 'database'/fact part and my facts are trying to be as logical as possible but its just not.
Quite exciting. Without diminishing the amazing value of LLMs, I don't think that path goes all the way to AGI. No idea if Carmack has the answer, but some good things will come out of that small research group, for sure.
I do agree that it is not particularly groundbreaking, but it's a nice "hey, here's our first update".
No comments yet
In a game there might be a level with a door and a key, and because there's no reward for getting the key closer to the door, bridging this gap requires random search in a massive state space. But in the vast sea of scenarios that you can find in Common Crawl there's probably one, where you are 1 step from the key, and the key is 1 step from the door, so you get the reward signal from it without having to search an enormous state space.
You might say "but you have to search through the giant Common Crawl". Well yes, but while doing so you will get reward signal not just for the key and door problem, but for nearly every problem in the world.
The point is: pretraining teaches models to extract signal that can be used to explore solutions to hard search problems, and if you don't do that you are wasting your time enumerating giant state spaces.
In this case, John is going off on this inane tangent because of his prior experience with hardware and video games instead of challenging himself to solve the actual hard and open problems.
I’m going to predict how this plays out for the inevitable screenshot in one to two years. John picks some existing RL algo and optimizes it to run in real time on real hardware. While he’s doing this the field moves on to better and new algorithms and architectures. John finally achieves his goal and posts a vid of some (now ancient) RL algo playing some Atari game in real time. Everyone says “neat” and moves on. John gets to feel validated yet all his work is completely useless.
John's document covers why he's doing what he's doing:
> Fundamentally, I believe in the importance of learning from a stream of interactive experience, as humans and animals do, which is quite different from the throw-everything-in-a-blender approach of pretraining an LLM. The blender approach can still be world-changingly valuable, but there are plenty of people advancing the state of the art there.
He thinks interacting with the real world and learning as you go isn't getting enough attention and might take us farther than the LLM approach. So he's applying these ideas to a subject that he's an expert in. You don't seem to find this approach interesting but John does (and I do too, for the record).
Everybody dismissing him might be right. Those keeping score know that Carmack's batting average isn't one thousand. But those people also know Carmack has the resources to work on pretty much whatever he wants to work on. I'm happy he's still working hard at something and sharing his work.
FWIW, I’m aware of that heuristic too which is why I intentionally use the word “just” as a meta heuristic to filter a certain type of person.
He’s a AAA software engineer but the prerequisites to build out cutting edge AI require deep formal math that is beyond his education and years at this point.
Nothing to stop him playing around with AI models though.
I'm pretty excited to see him in this domain. I think he'll focus on some DeepSeek style improvements.
Having JC focusing on, say, writing a performant OSS CUDA replacement could be bigger than any of the last 20 announcements from openai/goggle/deepmind/etc
So I asked Ilya, their chief scientist, for a reading list. This is my path, my way of doing things: give me a stack of all the stuff I need to know to actually be relevant in this space.
And he gave me a list of like 40 research papers and said, 'If you really learn all of these, you'll know 90% of what matters today! And I did. I plowed through all those things and it all started sorting out in my head.
To put it another way, the idea that John Carmack is going to do groundbreaking research in AI is roughly as plausible as the idea that Yann LeCun is going to make a successful AAA video game. Stranger things have happened, but I won’t be holding my breath.
In that context anyone can make progress in the field, as long as they understand what they're dealing with.
Better regard mr. Carmack as an X factor. Maybe the experts will leave him in the dust. Or maybe he'll come up with something that none of the experts cared to look into.
No comments yet
He's probably one of the most qualified people around.
All deep formal math is a boundary to a thing.
But don't get me wrong! Since this is a long-term research endeavor of his, I believe really starting from the basics is good for him and will empower him to bring something new to the table eventually.
I'm surprised though that he "only" came so far as of now. Maybe my slight idolization of Carmack made me kinda of blind to the fact that this kind of research is a mean beast after all and there is a reason that huuuuge research labs dump countless of man-decades into this kind of stuff with no guaranteed breakthroughs.
I'm nowhere as good at my craft as someone who works for openai, which the author of that tweet seems to be, but if even I can see this, then it's bad, isn't it?
No comments yet
It's still a lot better to really learn and discover it yourself to really get it.
Also it's hard to determine how much time someone spent on particular topic.
seems that we are learning in layers, one of the first layers being 2D neural net (images) augmented by other sensory data to create a 3D if not 4D model (neural net). HRTFs for sound increases the spatial data we get from images. With depth coming from sound and light and learnt movements(touch) we seem to develop a notion of space and time. (multimodality?)
Seems that we can take low dimensional inputs and correlate them to form higher dimensional structures.
Of course, physically it comes from noticing the dampening of visual data (in focus for example) and memorized audio data (sound frequency and amplitude, early reflections, doppler effect etc). That should be emergent from training.
Those data sources can be inperfectly correlated. That's why we count during a lightning storm to evaluate distance. It's low dimensional.
In a sense, it's a measure of required effort perhaps (distance to somewhere).
What's funny is that it seems to go the other way from traditional training where we move from higher dimensional tensor spaces to lower ones. At least in a first step.
Nonetheless, yes we do know certain brain structures like your image net analogy but the way you describe it, sounds a little bit of.
Our virtual cortex is not 'just a layer' its a component i would say and its optimized of detecting things.
Other components act differently with different structures.