Just like they "know" English.
"know" is quite an anthropomorphization. As long as an LLM will be able to describe what an evaluation is (why wouldn't it?) there's a reasonable expectation to distinguish/recognize/match patterns for evaluations. But to say they "know" is plenty of (unnecessary) steps ahead.
sidewndr46 · 3m ago
This was my thought as well when I read this. Using the word 'know' implies an LLM has cognition, which is a pretty huge claim just on its own.
noosphr · 19m ago
The anthropization of llms is getting off the charts.
They don't know they are being evaluated. The underlying distribution is skewed because of training data contamination.
zer00eyz · 5m ago
No, they do not. No LLM is ever going to be self aware.
It's a system that is trained, that only does what you build into. If you run an LLM for 10 years it's not going to "learn" anything new.
The whole industry needs to quit with the emergent thinking, reasoning, hallucination anthropomorphizing.
We have an amazing set of tools in LLM's, that have the potential to unlock another massive upswing in productivity, but the hype and snake oil are getting old.
They don't know they are being evaluated. The underlying distribution is skewed because of training data contamination.
It's a system that is trained, that only does what you build into. If you run an LLM for 10 years it's not going to "learn" anything new.
The whole industry needs to quit with the emergent thinking, reasoning, hallucination anthropomorphizing.
We have an amazing set of tools in LLM's, that have the potential to unlock another massive upswing in productivity, but the hype and snake oil are getting old.