Why evals are the unglamorous superpower

Everyone wants to improve their AI system. Almost no one can answer the prerequisite question: improve it according to what measure? Without evaluations — 'evals' — you're tuning prompts and swapping models based on vibes, unable to tell whether a change made things better or just different. Evals are the test suite for your AI. They're tedious to build and they're the single highest-leverage investment most teams aren't making.

Start from real failures

The temptation is to invent test cases from scratch. Don't start there. Start with reality:

• Collect actual failures. Every time your system gets something wrong — a bad answer, wrong format, a hallucinated fact, a missed edge case — capture it. These are gold. A failure that happened once will happen again.
• Capture the hard cases. The ambiguous inputs, the edge cases, the inputs that should produce a refusal or an 'I don't know.'
• Include the easy cases too, so you catch regressions where a change breaks something that used to work.

Your eval set should grow over time, accreting every new failure mode you discover. A six-month-old system with a rich eval set is far more improvable than a clever new one with none.

Decide what you're measuring

Different criteria need different judging methods. Sort what matters into three buckets:

Objective and checkable by code. Is the output valid JSON? Does it contain the required fields? Is it under the word limit? Does it cite a real source from the provided context? These you can test automatically and cheaply — write assertions, exactly like unit tests.

Factual against a known answer. For tasks with a correct answer — a classification, an extraction, a calculation — compare the output to the expected result. Be thoughtful about what counts as a match; trivial wording differences shouldn't fail a correct answer.

Subjective quality. Tone, helpfulness, clarity, whether an answer is genuinely good. These resist simple code checks and need human judgement — or a carefully-constructed model-as-judge.

Automatic checks first

For everything in the first two buckets, automate it. A script that runs your system against the eval set and reports pass/fail per case is the backbone. The payoff is enormous: you can run it after every change and instantly see what improved and what regressed. Many real-world quality issues are objective — wrong format, missing field, broke on empty input — and these are exactly what cheap automatic checks catch.

Judging the subjective stuff

For quality criteria that can't be coded, you have two main options.

Human evaluation is the gold standard. Have people rate outputs against a clear rubric — not 'is this good?' but specific, defined criteria scored consistently. The discipline is in the rubric: vague questions get noisy answers.

Model-as-judge uses a capable model to evaluate outputs against criteria you define. It's faster and cheaper than human review and scales well. The catch: the judge can be biased or wrong, so validate it against human judgement on a sample before trusting it, and write its instructions as carefully as any other prompt.

In practice, many teams use model-as-judge for routine scale and periodic human review to keep the judge honest.

Make evals part of the workflow

An eval set that lives in a folder no one runs is useless. To make it matter:

• Run evals before shipping any change — new prompt, new model, new retrieval logic. If the score drops, you've caught a regression before users did.
• Track scores over time so you can see the trajectory, not just the latest number.
• Set a bar for what's acceptable to ship, and hold to it.
• Add to the set whenever something new breaks.

A realistic starting point

Don't wait for the perfect eval framework — that's how teams end up with none. Start tiny: 20–30 real cases, a simple script that runs them and reports results, automatic checks for the objective stuff, and a quick human pass on the rest. That modest setup already puts you ahead of most teams, because it replaces 'I think this is better' with 'this scores higher on cases that matter.' Grow it from there, one failure at a time, and within months you'll have something that genuinely tells you whether your system is improving — which is the only way to reliably make it so.