Problems in LLM Benchmarking and Evaluation

Comments (4)

sbilstein · 3h ago

Personally I think LLM benchmarks make agents worse. All these companies chase the benchmarks, overfit, and think being able to cheat at the math olympiad is gonna get us to AGI. Instead researchers should peer in and get me an agent that can reliably count the number of "i"'s in mississippi.

upperhalfplane · 3h ago

I don't quite think they cheat at math olympiads, but obviously there are blindspots for the unspectacular tasks. That being said, Mississippi is both a good and a bad question to ask. On the one hand, it's "the bare minimum" to require, on the other hand, is it really a feat? Like, most models can write a piece of code that would compute that. If you show me a task I'm not designed to solve (like count the number of i's in this text), the smart thing is actually to write a program to count them (which LLMs can do).

The best way to measure intelligence is probably to have a model know its strengths and weaknesses, and deal with them in an efficient way. And the most important thing for eval is that ability.

tianlong · 3h ago

What's the TLDR about how to solve that benchmark problem?

upperhalfplane · 3h ago

TLDR: games are a good way to go.

VC-backed company just killed my EU trademark for a small OSS project

F-Droid build servers can't build modern Android apps due to outdated CPUs

Ask HN: How do you tune your personality to get better at interviews?

Ask HN: What are your favorite obscure but brilliant C/C++ libraries?

Ask HN: What's your most valuable query to an LLM?

Ask HN: How are you scaling AI agents reliably in production?

One weird trick to making Claude Code palatable to use

Individual Bestbuy email subscription pages are apparently indexed by Google

What would you name a new programming language?

Ask HN: How is Cognition able to raise at $10B?

Ask HN: Is the rise of AI tools going to be the next 'dot com' bust?

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

Ask HN: What alternatives to GitHub are you using?

Ask HN: What "developer holy war" have you flip-flopped on?

Ask HN: Do you use personal AI Agents?

Ask HN: What toolchains are people using for desktop app development in 2025?

Ask HN: Is Twilio necessary and required to build text and voice agents?

Ask HN: How do you connect with other founders in your city?

Ask HN: Are there software engineering areas that are safe from LLMs invasion?

Ask HN: What's stopping Guix from building static binaries for various targets?

Ask HN: Is there an AI that can read code aloud and explain it?

Ask HN: How do you like your RSS feed?

Ask HN: Why aren't more developers using AI tools?

What is the learning curve for using generative AI effectively?

My inbox is a mess from signing up for everything. How do you handle it?

Best area of AI to focus on for a front end developer?

Ask HN: Why Is My Happiness Tied to My Productivity?

Ask HN: With all the AI hype, how are software engineers feeling?

Gemini's brutal assessment of a vibe coding session

Problems in LLM Benchmarking and Evaluation

Comments (4)