A new model drops. The announcement has a bar chart where it edges out the competition on a row of benchmarks. Should you switch? The honest answer is that the chart tells you almost nothing about whether the model is right for you. Here's a practical framework for making that call properly.

Why public benchmarks mislead

Benchmarks aren't useless, but they're badly overweighted in most decisions, for concrete reasons:

Contamination. Many benchmark questions and answers have leaked into training data. A model can score well by having effectively seen the test, not by being more capable. This is hard to detect and pervasive.

Gaming. When a benchmark becomes a marketing target, models get tuned to it specifically. A high score may reflect optimisation toward that test rather than general capability.

Relevance. A benchmark measures performance on its tasks. Competition maths scores tell you little about how a model drafts your customer emails or extracts fields from your invoices. The correlation between benchmark rank and your-task performance is weaker than people assume.

Saturation. On many benchmarks the top models are clustered within a point or two, well inside the noise. "Beats the rival by 1.3 points" is often meaningless.

Build your own eval

The single most valuable thing you can do is build a small, private evaluation set from your actual use case. It doesn't need to be fancy:

1. Collect 20–100 real examples of the task you care about — real prompts, real inputs, drawn from production if possible.
2. Define what 'good' means for each. Sometimes there's a correct answer; often it's a rubric (did it stay on brand, extract the right fields, avoid the forbidden phrasing).
3. Run every candidate model through the set under identical conditions.
4. Score the outputs — automatically where you can, by human judgement where you must. Even rough scoring beats a leaderboard.

This private eval is reusable: every time a new model launches, run it through and get a real answer in an afternoon. It's the highest-leverage piece of evaluation infrastructure most teams never build.

Look past quality alone

Raw output quality is one dimension. A model can win on quality and still be the wrong choice:

Latency. How fast does it respond on your prompt sizes? A slightly better but noticeably slower model can hurt an interactive product more than the quality gain helps.

Cost. Price per token, multiplied by your real token volumes including any hidden reasoning tokens. A 'better' model that costs several times more may not be worth it for your task mix.

Reliability and consistency. Does it return well-formed output every time, or does it occasionally break format and crash your parser? For automated pipelines, consistency often matters more than peak quality.

Failure modes. How does it fail? A model that says "I don't know" when unsure is far safer in many products than one that fails confidently and plausibly. Test deliberately on inputs near its competence edge.

Test the things that bite

Go beyond the happy path:

• Adversarial and malformed inputs — does it degrade gracefully or fall apart?
• Instruction adherence under pressure — does it still follow your format rules on hard or unusual inputs?
• Edge cases from your domain — the weird, rare inputs that production will eventually throw at it.

A sane process

Put together, a good evaluation looks like this: use public benchmarks only as a coarse filter to assemble a shortlist; build a private eval from your real tasks; run the shortlist through it; layer in latency, cost, reliability, and failure-mode testing; then decide. The whole thing can take a day and will serve you for every future model launch.

The model with the tallest bar on launch day is not necessarily the model that will serve your users best. The only way to know is to test it on your work, with your standards, measuring the things your product actually depends on.