LLM Evals Are Just Tests. Why Are We Making This So Complicated?

3 camwest 2 8/10/2025, 3:23:48 AM cameronwestland.com ↗

Comments (2)

8organicbits · 2d ago
So, did the tests allow you to build a system that never confused existing features with new features? That seems like the problem statement, but I think I'm only seeing probabilistic testing.
camwest · 1d ago
Never? No. Way less likely? Yes!

In dev we do 100 consistency checks and get green. In CI we do 10.