Language model benchmarks only tell half a story