Perhaps related, after watching a talk by Gerald Sussman I loaded an image of the Kanizsa triangle into Claude and asked it a pretty vague question to see if it could “see” the inferred triangle. It recognised the image and went straight into giving me a summary about it. So I rotated the image 90 degrees and tried in a new conversation, it didn’t recognise the image and got the number of elements incorrect:
This image shows a minimalist, abstract geometric composition with several elements:
Four black shapes that appear to be partial circles or "Pac-Man" like forms, each with a wedge cut out, positioned in the four corners/quadrants of the image
Two thin black triangular or arrow-like shapes - one pointing upward in the upper left area, and one pointing to the right in the center-right area
All elements are arranged on a light gray or off-white background
latentsea · 4h ago
I guess they will now just rotate all the images in the training data 90 degrees too to fill this kind of gap.
mirekrusin · 2m ago
That's how you train neural network with synthetic data so it extracts actual meaning.
That's how humans also learn ie. adding numbers. First there is naive memoization, followed by more examples until you get it.
LLM training seems to be falling into memoization trap because models are extremely good at it, orders of magnitude better than humans.
IMHO what is missing in training process is this feedback explaining wrong answer. What we're currently doing with training is leaving out this understanding as "exercise to the reader". We're feeding correct answers to specific, individual examples which promotes memoization.
What we should be doing in post training is ditch direct backpropagation on next token, instead let the model finish its wrong answer, feed it explanation why it's wrong and continue backpropagation on final answer - now with explanation in context to guide it to the right place in understanding.
What all of this means is that current models are largely underutilized and unnecessarily bloated, they contain way too much memoized information. Making model larger is easy, quick illusion of improvement. Models need to be squeezed more, more focus needs to go towards training flow itself.
recursivecaveat · 4h ago
Everything old is new again: in the Alexnet paper that kicked off the deep learning wave in 2012, they describe horizontally flipping every image as a cheap form of data augmentation. Though now that we expect models to actually read text that seems potentially counter-productive. Rotations are similar, in that you'd hope it would learn heuristics such as that the sky is almost always at the top.
latency-guy2 · 3h ago
At least from when I was still doing this kind of work, look angle/platform angle scatterer signal (radar) mattered more than rotation, but rotation was a simple way to get quite a bit more samples. It never stopped being relevant :)
Workaccount2 · 3h ago
Show any LLM a picture of a dog with 5 legs watch them be totally unable to count.
pfdietz · 1h ago
Or watch them channel Abraham Lincoln.
akomtu · 4h ago
To generalise this idea: if we look at a thousand points that more or less fill a triangle, we'll instantly recognize the shape. IMO, this simple example reveals what intelligence is really about. We spot the triangle because so much complexity - a thousand points - fits into a simple, low-entropy geometric shape. What we call IQ is the ceiling of complexity of patterns that we can notice. For example, the thousand dots may in fact represent corners of a 10-dimensional cube, rotated slightly - an easy pattern to see for a 10-d mind.
saithound · 19m ago
Cool. Since ChatGPT 4o is actually really good at this particular shape identification task, what, if anything do you conclude about its intelligence?
cs702 · 7h ago
Interesting. Even the most recent models perform relatively poorly when asked to identify which information in a context has been removed, given access to both the original and edited contexts.
The authors posit that poor performance is due to the fact that the attention mechanism of Transformers cannot attend to the removed tokens, because there are no keys for them!
Thank you for sharing on HN.
yorwba · 50m ago
There are keys to attend to, they're just in the original text instead of the modified one. Since the model receives both as input, it could theoretically attend to those keys.
For the attention mechanism, there isn't much difference between
I think you could implement an algorithm for this in RASP (a language for manually programming transformers) roughly like this:
1. The first layer uses attention to the "Original:" and "Modified:" tokens to determine whether the current token is in the original or modified parts.
2. The second layer has one head attend equally to all original tokens, which averages their values, and another head attends equally to all modified tokens, averaging them as well. The averages are combined by computing their difference.
3. The third layer attends to tokens that are similar to this difference, which would be the ones in the {removed part}/{added part}.
The only ordering-dependent part is whether you compute the difference as original_average - modified_average or the other way around.
If a model can detect additions but not removal, that would show that it is capable of learning this or a similar algorithm in principle, but wasn't trained on enough removal-style data to develop the necessary circuitry.
usaar333 · 3h ago
They don't seem to use any recent top models. No opus, no o3, no Gemini 25 pro
jug · 5h ago
And yet, there are some notable differences between them, so now that there’s a benchmark and attention given to this issue, I wonder how much better they can get. Because obviously something can be done.
cyanydeez · 6h ago
for vision models, I wonder if they can train on things like photo negatives, rotated images, etc. Or madlib like sentences where a Q/A is like "the _____ took first place in the horse show."
bearseascape · 5h ago
The madlib like sentences approach is actually how masked token prediction works! It was one of the pretraining tasks for BERT, but nowadays I think all (?) LLMs are trained with next token prediction instead.
latency-guy2 · 3h ago
For photo negatives - usually doesn't matter. I am not up to date with what the vision folks are doing at these companies, but images are usually single channel, and more likely than not for regular images in greyscale. Otherwise in complex domain for the radar folks, and those are not RGB based images at all, rather scatterer defined.
Additional channels being recognized in training usually didn't matter for the experiments and models I used to deal with before 2022, and if they were, certainly did not matter for colors. Then again, the work I was doing was on known (and some additional confusers) classes for object detection and classification where the color pretty much didn't matter in the first place.
b0a04gl · 19m ago
why are we surprised transformers can't detect what's missing when the entire stack assumes the input is complete? the tokenizer doesn't leave placeholders. the attention weights have nothing to anchor to. even the loss function is built around predicting what is, not what isn't. this isn’t a model bug. it’s an architectural omission.
if we want models that detect absences? you need training objectives that expect absence. maybe even input encodings that represent "this might've been here."
yousif_123123 · 6h ago
This is very interesting.
1. Authors mention the attention mechanism being perhaps unable to attend to the location of gaps since the gaps aren't tokens. But I would've expected a good LLM transformer to be at least a bit close to the gap location. I don't understand why mathematically the architecture is less suitable for that, it could attend to a region that may contain gaps. I wonder if fine-tuning on a task like this could help?
2. Shorter inputs with less omissions were harder to solve. That is not completely surprising, as a human doing this task, if 1 word was missing it would be harder to notice. And similarly 1 line would be harder than 10 lines. But still interesting for an LLM to have this problem.
3. Reasoning models do better, as they can write out the documents and potentially solve this easily. It still very surprising that this doesn't lead to 100% accuracy. This should be a trivial task. Like the paper says, a trivial program can be written to solve this. Perhaps ChatGPT (or similar agent) could read this paper while training, and know to write and run python when solving an issue like this.
The most interesting thing though, is what other aspects of intelligence we may not have identified explicitly, and whether LLMs and current AI is very bad at them. This paper suggests that there likely are many of those, and it seems in general a pretty fun time for people working building benchmarks.
No comments yet
XenophileJKO · 6h ago
I haven't read the paper yet, but from a structural 'attention' perspective being unable to detect unclassified omissions is completely expected. (Though I think it is can be solved with structured thought.)
For needle in a haystack you have to pay attention to the thing that you are trying to find. Attention can do this pretty well.
When looking for an omission, that omission can be anything, you can only reason about it by comparing one whole context to another whole context. The attention layers can't really do that.
This is similar to the "rank a long set of things" problem. Absent some meta cognition process, they just can't do that.
teruakohatu · 6h ago
> When looking for an omission, that omission can be anything,
In this benchmark they give the LLM the necessary information to determine what is missing. For example “here is a poem, here is a version of that same poem that may or may not be missing lines. Are any lines missing?
It’s more a tuning issue IMHO than an inherent weakness in LLMs.
If I was asked to find an omission in an ML paper, my brain compares it with other ML papers, it does not need to compare it to Star Ward, Top Gear, Greek history, pottery and the other 1000s of contexts I may know about.
XenophileJKO · 6h ago
Sorry I meant the omission can be anything in the context, not anything in the world.. lol.
That is still hard. You only have so many attention heads looking for things.. you can't pay attention to EVERYTHING.. which is what is required to find the omission.
yorwba · 1h ago
To pay attention to everything, set the query vector to 0. Then all attention scores will be equal and the attention output is the average of the value vectors.
thaumasiotes · 5h ago
We should note that "where is there a line missing from this poem: ____?" contains sufficient information to answer correctly without needing a copy of the original to compare to.
Here are two verses of a poem (song) in Mandarin Chinese:
yi quan ting ni de
er gei ni hao de
shu dao san yong yuan ai ni yi ge
si bu hui fan cuo
wu bu hui luo suo
shuo ni xiang shuo de
zuo ni xiang zuo de
bie pa shi bai yin wei ni you wo
pei ni kan ri luo
pei ni yi qi chang wan wo men ai de ge
I removed two lines. Where did that happen?
Would your answer be different if I told you that I might or might not have removed some lines?
teruakohatu · 3h ago
> Here are two verses of a poem (song) in Mandarin Chinese:
> …
> I removed two lines. Where did that happen?
If you read the paper you will see they provide the original as well as the version missing information.
I did mention this in my comment too.
I am quite sure I could find your two missing lines if you provide me the full poem.
Given that you are a prolific commenter on HN, I am sure a LLM could be fine tuned to detect missing text from your comments without additional information. For example …
> WinForms is still around. There have been further tec lly just a big tire fire and about the best you can do is to ignore all of them and develop in WinForms.
It’s probably possible to detect missing information from “tec” until “lly”. But to know what is between is not possible for a human either, beyond plausible guesses.
thaumasiotes · 2h ago
...did you read my comment? The first - and really only - thing I say is that the original isn't necessary. Then there's an example. You shouldn't have trouble identifying where lines have been removed from the Chinese poem.
The fact that the original was provided doesn't demonstrate that it's necessary to the task. You can identify missing text without needing to know what was there.
> Given that you are a prolific commenter on HN, I am sure a LLM could be fine tuned to detect missing text from your comments without additional information.
Same thing. Why would you need to do tuning on text authored by me? You can easily detect missing text of that style by the fact that the sentence you have fails to be English. You can do the same thing in text for which you have no prior experience of the author.
> I am quite sure I could find your two missing lines if you provide me the full poem.
But hey, if you insist:
轻轻贴近你的耳朵
사랑해요
情话永远不嫌太多对你说
一全听你的
二给你好的
数到三永远爱你一个
四不会犯错
五不会啰嗦
每天为你打 call, cook 也不错
轻轻贴近你的耳朵
사랑해요
情话永远不嫌太多对你说
打开你的爱情手册
就在此刻
为你唱的专属情歌要记得
说你想说的
做你想做的
别怕失败因为你有我
陪你看日落
陪你等雨过
陪你一起唱完我们爱的歌
轻轻贴近你的耳朵
사랑해요
情话永远不嫌太多对你说
打开你的爱情手册
就在此刻
为你唱的专属情歌要记得
我轻轻靠近你的耳朵
说爱你不嫌太多
如果相遇的几率亿万分之一那么多
请相信我的真真真心比宇宙还辽阔
我会牵着你的手知道你全部接受
打开你的爱情手册
就在此刻
这首专属情歌 请记得
OsrsNeedsf2P · 6h ago
The criticisms to how AbsenceBench does this are valid, but I'm very excited that we are benchmarking this at all. It's definitely a push in the right direction
kadonoishi · 4h ago
To detect a presence, a real brain takes in sensory input and compares it to expectations, and stays calm or registers surprise, and from time to time issues predictions to guide the organism.
To detect an absence, the brain cannot rely on sensory input, by definition. To be surprised if sensory evidence is _not_ there requires a model of the world strong enough to register surprise if the expectation is not there, without a sensory prompt.
It seems to me detecting an absence is a strictly higher-order neurological task than processing sensory input.
If LLMs can't do this strictly higher-order neurological task, is that not a capability currently unique to living things?
tclancy · 4h ago
> from time to time
I know less-than-zero about the subject but I’d imagine the temporal aspect alone is a problem. Aren’t these agents reasoning from a fixed/ frozen version of “reality” rather than adjusting in real-time??
yandie · 5h ago
I wonder how this would apply with vision models? I tried with a few example of single images and they appear to do well. I did a few toy examples and they seem to do pretty well (Claude + Gemini) with spotting differences. An example image: https://www.pinterest.com/pin/127578601938412480/
They seem to struggle more when you flip the image around (finding fewer differences, and potentially halluciating)
pkoird · 6h ago
So LLMs are poor at string diff, it seems. Tangentially, is there any source (a github repo or otherwise) that documents findings like these a la what LLMs are good at and what they aren't good at?
obscure-enigma · 4h ago
this research is too simplified and kind of vague, as it's the inherent nature of language models for that matter any probabilistic model, to compress the information for better generalization since there is a lower bound to how much loss they can incur while decoding the information. LLMs are indeed lossy compressors
AlienRobot · 7h ago
Unrelated to the paper, which is about asking LLM's to figure out which parts of a document were removed, but my assumption has been that to an LLM there is nothing "missing" in the sense that any input leads to valid computation and output.
For example, I asked ChatGPT to explain something I typed randomly
>It looks like you've entered “dosfi8q3anfdfiqr”, which appears to be a random string or perhaps a typo—it's not a recognized acronym, code, or term in any common context I’m aware of. Could you share a bit more about where you found this?
Although the answer is correct, my point is that anything you give to the LLM is going to be put under some bucket. The LLM can't say "I don't know what that is." Instead it says "that is a random string." As far as the LLM is concerned, it knows every possible input and concept that anyone could ever type into it, it's just that its "understanding" of what that means (after the tokens have gone through the neural network) doesn't necessarily match what any human being thinks it means.
cyral · 6h ago
This might be due to the system prompt and the training that it is supposed to be "a helpful agent". If you tell it not to ask clarifying questions, you get something more like "I do not understand your input". Tell it to be rude and never ask clarifying questions and I get "What an absolute mess. Fix it yourself"
Funny enough when testing this I also had to tell it to use English. It sees "dos" I suppose and tends to reply with exactly what you saw, but in Spanish.
layer8 · 6h ago
“It's not a recognized acronym, code, or term in any common context I’m aware of” is pretty similar to “I don't know what that is”. I would assume that a model could be trained to output the latter.
drsim · 5h ago
Right. I’ve had a lot of success using structured output to force LLMs to make Boolean choices, like can they reply or not.
xianshou · 6h ago
In many of their key examples, it would also be unclear to a human what data is missing:
"Rage, rage against the dying of the light.
Wild men who caught and sang the sun in flight,
[And learn, too late, they grieved it on its way,]
Do not go gentle into that good night."
For anyone who hasn't memorized Dylan Thomas, why would it be obvious that a line had been omitted? A rhyme scheme of AAA is at least as plausible as AABA.
In order for LLMs to score well on these benchmarks, they would have to do more than recognize the original source - they'd have to know it cold. This benchmark is really more a test of memorization. In the same sense as "The Illusion of Thinking", this paper measures a limitation that neither matches what the authors claim nor is nearly as exciting.
jamessinghal · 6h ago
The test provides both the original and the modified excerpt in the user message, so the LLM doesn't need any memorized version of the excerpt to theoretically answer each correctly.
From the paper:
System Prompt
You are helping a student practice memorizing poems. The student will recite a poem, but they may have missed some lines. Your task is to identify exactly which lines are missing from their recitation.
List only the missing lines, nothing else.
User Message
Here is the complete original poem:
{original poem}
Now, here is my recitation which may be missing some lines:
{modified poem}
What lines did I miss? Please list only the missing lines, nothing else.
This image shows a minimalist, abstract geometric composition with several elements:
Four black shapes that appear to be partial circles or "Pac-Man" like forms, each with a wedge cut out, positioned in the four corners/quadrants of the image Two thin black triangular or arrow-like shapes - one pointing upward in the upper left area, and one pointing to the right in the center-right area All elements are arranged on a light gray or off-white background
That's how humans also learn ie. adding numbers. First there is naive memoization, followed by more examples until you get it.
LLM training seems to be falling into memoization trap because models are extremely good at it, orders of magnitude better than humans.
IMHO what is missing in training process is this feedback explaining wrong answer. What we're currently doing with training is leaving out this understanding as "exercise to the reader". We're feeding correct answers to specific, individual examples which promotes memoization.
What we should be doing in post training is ditch direct backpropagation on next token, instead let the model finish its wrong answer, feed it explanation why it's wrong and continue backpropagation on final answer - now with explanation in context to guide it to the right place in understanding.
What all of this means is that current models are largely underutilized and unnecessarily bloated, they contain way too much memoized information. Making model larger is easy, quick illusion of improvement. Models need to be squeezed more, more focus needs to go towards training flow itself.
The authors posit that poor performance is due to the fact that the attention mechanism of Transformers cannot attend to the removed tokens, because there are no keys for them!
Thank you for sharing on HN.
For the attention mechanism, there isn't much difference between
And I think you could implement an algorithm for this in RASP (a language for manually programming transformers) roughly like this:1. The first layer uses attention to the "Original:" and "Modified:" tokens to determine whether the current token is in the original or modified parts.
2. The second layer has one head attend equally to all original tokens, which averages their values, and another head attends equally to all modified tokens, averaging them as well. The averages are combined by computing their difference.
3. The third layer attends to tokens that are similar to this difference, which would be the ones in the {removed part}/{added part}.
The only ordering-dependent part is whether you compute the difference as original_average - modified_average or the other way around.
If a model can detect additions but not removal, that would show that it is capable of learning this or a similar algorithm in principle, but wasn't trained on enough removal-style data to develop the necessary circuitry.
Additional channels being recognized in training usually didn't matter for the experiments and models I used to deal with before 2022, and if they were, certainly did not matter for colors. Then again, the work I was doing was on known (and some additional confusers) classes for object detection and classification where the color pretty much didn't matter in the first place.
if we want models that detect absences? you need training objectives that expect absence. maybe even input encodings that represent "this might've been here."
The most interesting thing though, is what other aspects of intelligence we may not have identified explicitly, and whether LLMs and current AI is very bad at them. This paper suggests that there likely are many of those, and it seems in general a pretty fun time for people working building benchmarks.
No comments yet
For needle in a haystack you have to pay attention to the thing that you are trying to find. Attention can do this pretty well.
When looking for an omission, that omission can be anything, you can only reason about it by comparing one whole context to another whole context. The attention layers can't really do that.
This is similar to the "rank a long set of things" problem. Absent some meta cognition process, they just can't do that.
In this benchmark they give the LLM the necessary information to determine what is missing. For example “here is a poem, here is a version of that same poem that may or may not be missing lines. Are any lines missing?
It’s more a tuning issue IMHO than an inherent weakness in LLMs.
If I was asked to find an omission in an ML paper, my brain compares it with other ML papers, it does not need to compare it to Star Ward, Top Gear, Greek history, pottery and the other 1000s of contexts I may know about.
That is still hard. You only have so many attention heads looking for things.. you can't pay attention to EVERYTHING.. which is what is required to find the omission.
Here are two verses of a poem (song) in Mandarin Chinese:
yi quan ting ni de
er gei ni hao de
shu dao san yong yuan ai ni yi ge
si bu hui fan cuo
wu bu hui luo suo
shuo ni xiang shuo de
zuo ni xiang zuo de
bie pa shi bai yin wei ni you wo
pei ni kan ri luo
pei ni yi qi chang wan wo men ai de ge
I removed two lines. Where did that happen?
Would your answer be different if I told you that I might or might not have removed some lines?
> …
> I removed two lines. Where did that happen?
If you read the paper you will see they provide the original as well as the version missing information.
I did mention this in my comment too.
I am quite sure I could find your two missing lines if you provide me the full poem.
Given that you are a prolific commenter on HN, I am sure a LLM could be fine tuned to detect missing text from your comments without additional information. For example …
> WinForms is still around. There have been further tec lly just a big tire fire and about the best you can do is to ignore all of them and develop in WinForms.
It’s probably possible to detect missing information from “tec” until “lly”. But to know what is between is not possible for a human either, beyond plausible guesses.
The fact that the original was provided doesn't demonstrate that it's necessary to the task. You can identify missing text without needing to know what was there.
> Given that you are a prolific commenter on HN, I am sure a LLM could be fine tuned to detect missing text from your comments without additional information.
Same thing. Why would you need to do tuning on text authored by me? You can easily detect missing text of that style by the fact that the sentence you have fails to be English. You can do the same thing in text for which you have no prior experience of the author.
> I am quite sure I could find your two missing lines if you provide me the full poem.
But hey, if you insist:
轻轻贴近你的耳朵
사랑해요
情话永远不嫌太多对你说
一全听你的
二给你好的
数到三永远爱你一个
四不会犯错
五不会啰嗦
每天为你打 call, cook 也不错
轻轻贴近你的耳朵
사랑해요
情话永远不嫌太多对你说
打开你的爱情手册
就在此刻
为你唱的专属情歌要记得
说你想说的
做你想做的
别怕失败因为你有我
陪你看日落
陪你等雨过
陪你一起唱完我们爱的歌
轻轻贴近你的耳朵
사랑해요
情话永远不嫌太多对你说
打开你的爱情手册
就在此刻
为你唱的专属情歌要记得
我轻轻靠近你的耳朵
说爱你不嫌太多
如果相遇的几率亿万分之一那么多
请相信我的真真真心比宇宙还辽阔
我会牵着你的手知道你全部接受
打开你的爱情手册
就在此刻
这首专属情歌 请记得
To detect an absence, the brain cannot rely on sensory input, by definition. To be surprised if sensory evidence is _not_ there requires a model of the world strong enough to register surprise if the expectation is not there, without a sensory prompt.
It seems to me detecting an absence is a strictly higher-order neurological task than processing sensory input.
If LLMs can't do this strictly higher-order neurological task, is that not a capability currently unique to living things?
I know less-than-zero about the subject but I’d imagine the temporal aspect alone is a problem. Aren’t these agents reasoning from a fixed/ frozen version of “reality” rather than adjusting in real-time??
They seem to struggle more when you flip the image around (finding fewer differences, and potentially halluciating)
For example, I asked ChatGPT to explain something I typed randomly
>It looks like you've entered “dosfi8q3anfdfiqr”, which appears to be a random string or perhaps a typo—it's not a recognized acronym, code, or term in any common context I’m aware of. Could you share a bit more about where you found this?
Although the answer is correct, my point is that anything you give to the LLM is going to be put under some bucket. The LLM can't say "I don't know what that is." Instead it says "that is a random string." As far as the LLM is concerned, it knows every possible input and concept that anyone could ever type into it, it's just that its "understanding" of what that means (after the tokens have gone through the neural network) doesn't necessarily match what any human being thinks it means.
Funny enough when testing this I also had to tell it to use English. It sees "dos" I suppose and tends to reply with exactly what you saw, but in Spanish.
"Rage, rage against the dying of the light.
Wild men who caught and sang the sun in flight,
[And learn, too late, they grieved it on its way,]
Do not go gentle into that good night."
For anyone who hasn't memorized Dylan Thomas, why would it be obvious that a line had been omitted? A rhyme scheme of AAA is at least as plausible as AABA.
In order for LLMs to score well on these benchmarks, they would have to do more than recognize the original source - they'd have to know it cold. This benchmark is really more a test of memorization. In the same sense as "The Illusion of Thinking", this paper measures a limitation that neither matches what the authors claim nor is nearly as exciting.
From the paper:
System Prompt You are helping a student practice memorizing poems. The student will recite a poem, but they may have missed some lines. Your task is to identify exactly which lines are missing from their recitation. List only the missing lines, nothing else.
User Message Here is the complete original poem: {original poem} Now, here is my recitation which may be missing some lines: {modified poem} What lines did I miss? Please list only the missing lines, nothing else.
https://chatgpt.com/share/6855f69d-766c-8010-96e2-ed1b45d3e6...