Every AI model launch arrives with a chart. A cluster of bars, the new model's bar tallest, a rival's just behind, and a number — 92.4%, say — presented as proof that the thing is now the best. The chart is not a lie, exactly. But it is a sales document, and learning to read it the way a skeptic would is one of the most useful habits in following AI. Here is how model scoring actually works, and why the number on the slide is almost never the whole story.

What a benchmark is

A benchmark is a fixed test: a set of questions with known answers, and a rule for scoring. Run the model over the set, count what it gets right, and you have a percentage you can put on a chart. The best-known ones each probe a different skill. MMLU is a broad multiple-choice exam across dozens of subjects. GPQA is deliberately hard graduate-level science, written to be "Google-proof." SWE-bench hands the model real GitHub issues that were resolved by human pull requests, and checks whether its own patch makes the project's tests pass. AIME and MATH measure competition mathematics. And LMArena (formerly Chatbot Arena) does something different: it shows anonymous people two models' answers side by side and asks which they prefer, then ranks models by an Elo-style rating like chess players.

Taken together these are useful. A model that jumps twenty points on SWE-bench really can write better code. The problem is not that benchmarks are meaningless — it is that the number is far softer than it looks, for four reasons that recur constantly.

Why the number is softer than it looks

1. Contamination. This is the big one. If a benchmark's questions and answers have leaked into a model's training data — and after years of scraping the web, many have — then a high score can reflect memorisation rather than reasoning. The model isn't solving the problem; it has seen the answer key. Labs increasingly test on freshly written or held-out questions precisely because scores on older public benchmarks have become hard to trust.

2. Goodhart's Law. The economist's adage — "when a measure becomes a target, it ceases to be a good measure" — is the central curse of AI evaluation. Once a benchmark matters commercially, labs optimise for it directly: fine-tuning on similar problems, tuning prompts, tweaking output formats. The score goes up; the underlying capability may not move nearly as much. This is the same failure we described in how models get their manners as reward hacking — a system optimising the proxy instead of the goal.

3. Cherry-picking and configuration. A vendor chooses which benchmarks appear on the slide, and picks the ones it wins. It also picks the setup — a model run with maximum reasoning effort, multiple attempts, or a bespoke prompt can post a number that an ordinary user will never see. Two "SWE-bench" figures can mean quite different things depending on the scaffolding around the model. When a comparison shows a model beating rivals, the honest question is always: measured how, and against which version?

4. Saturation and validity. As top models cluster near the ceiling on a test, that test stops telling them apart — MMLU, where the best models now sit around 90%, is largely retired as a frontier signal for this reason. And a benchmark only matters if it measures what you care about: a set of self-contained coding puzzles is not the same as maintaining a large, messy codebase, however high the score. The measure can be real and still not map to the job.

The human-preference alternative — and its own trap

Because static benchmarks are so gameable, a lot of attention has shifted to LMArena-style human voting. It is harder to game, because you cannot memorise a preference. But it has its own bias: people tend to prefer answers that are longer, more confident, and formatted attractively, which rewards models for sounding good over being right — the sycophancy problem again, wearing a leaderboard as a hat. A model can climb the arena by being agreeable and verbose. Preference is a signal, not a verdict, either.

How to read the chart

None of this means benchmarks are worthless. It means they are evidence, not proof, and the skeptic's questions are simple:

• Is the number independently verified, or is it the vendor's own claim? This is why careful coverage — ours included — labels benchmark figures as "vendor-reported" until a third party reproduces them.
• Could the test be contaminated? Newer, private, or held-out benchmarks are worth more than old public ones.
• Measured in what configuration — and would a normal user ever run the model that way?
• Does the benchmark resemble your task? A model that tops a leaderboard can still lose on the one job you actually need it for — as one startup showed this month when a $500 fine-tune beat five frontier models on a single narrow task it was built for.

That last point is the one that matters most in practice. The only evaluation that fully counts is the one you run yourself, on your own data, for your own use — a small, private test set of the tasks you care about will tell you more than any launch-day chart. The public benchmarks set the weather. The eval you build tells you whether it will rain on you.

The short version:

• A benchmark is a fixed test with a score — MMLU, GPQA, SWE-bench, AIME, and preference arenas like LMArena each measure something different and are all useful.
• The number is softer than it looks because of contamination (answers leaking into training data), Goodhart's Law (optimising the measure, not the skill), cherry-picked benchmarks and configurations, and saturation.
• Human-preference leaderboards are harder to game but reward length, confidence and style — sounding good over being right.
• Treat any benchmark as a vendor claim until independently verified, and remember the only eval that truly counts is the one you run on your own task.