I think they are basically right, but it's not at the level of "test taking" it's at the level of "linguistic competence".
I worked on a few projects that tried to develop foundation models and ruled out or tried to rule out [1] quite a few approaches based on arguments like "in the tokenization step you lose critical information in 8% of all cases so that puts a ceiling of 92% accuracy"
That wasn't quite right because that's assuming the model has to get the right answer by the right process, if you give it credit for the right answer by the wrong process then maybe it makes a wild assed guess which is right 50% of the time so the ceiling is more like 96%. But we call that last 4% a "hallucination".
You could make the case that we could make models that do a better job of reasoning about probability but I think that the magic of LLMs is that they reason about probability wrong in a way that empirically works and it's my perennially unpopular opinion that the "language instinct" in humans is similarly a derangement of reasoning about probability that collapses the manifold of possible productions into a lower-dimensional space which is easier to learn.
[1] properly in the case of those models I think because nobody else has made progress with then
I worked on a few projects that tried to develop foundation models and ruled out or tried to rule out [1] quite a few approaches based on arguments like "in the tokenization step you lose critical information in 8% of all cases so that puts a ceiling of 92% accuracy"
That wasn't quite right because that's assuming the model has to get the right answer by the right process, if you give it credit for the right answer by the wrong process then maybe it makes a wild assed guess which is right 50% of the time so the ceiling is more like 96%. But we call that last 4% a "hallucination".
You could make the case that we could make models that do a better job of reasoning about probability but I think that the magic of LLMs is that they reason about probability wrong in a way that empirically works and it's my perennially unpopular opinion that the "language instinct" in humans is similarly a derangement of reasoning about probability that collapses the manifold of possible productions into a lower-dimensional space which is easier to learn.
[1] properly in the case of those models I think because nobody else has made progress with then