Language Models Improve When Pretraining Data Matches Target Tasks

Isn’t this just benchmarks? Even if they claim the gains generalize, it still seems like they're only optimizing for benchmarks, which is not all there is. Unless benchmark scores perfectly correlate with real-world performance (which they likely don’t), it's not clear this helps much in practice.