The Eval Comes Before the Purchase

The standard way teams pick a model takes less than an hour: open a public leaderboard, sort by the headline score, shortlist the top few, and start integrating the one that fits the budget. It is fast, defensible in a meeting, and answers a question you do not have.

A leaderboard measures how a model performs on a fixed public benchmark — exam questions, math problems, crowd-sourced preference votes. Your product does not answer exam questions. It classifies your tickets, extracts your fields, drafts in your voice, and reasons over your documents. The gap between benchmark performance and workload performance is not a rounding error; it is routinely wide enough to flip rankings. A model that dominates the aggregate board can lose on your task to a smaller model that happens to be strong at exactly your shape of work.

This is the first post in this cluster that asks you to spend effort before money: build the evaluation first, buy second. The arithmetic of what a passing model costs is covered in cost per completed task; this post is about the scoreboard that arithmetic depends on.

The raw material is already in your logs. Pull a slice of real production requests — a few hundred is enough to start, more if the workload is diverse — and strip anything sensitive. What you want is not a sample of easy cases but a faithful cross-section: the routine majority plus the long tail that actually breaks models. If your traffic splits into distinct task shapes, sample each shape separately so one dominant case does not drown the rest.

For each example, record the input exactly as the model would see it and what a correct answer looks like. "Correct" does not have to mean a single golden string. For extraction and classification it can be exact match; for drafting it is usually a rubric — the facts that must appear, the format that must hold, the tone that must not slip. Rubric-graded answers take longer to write down, and they are the difference between an eval and a demo.

Two disciplines pay off immediately. Version the set, because you will rerun it for years and need to know what changed. And include the failures you already know about: the prompt that broke the last model is worth more than ten prompts that never broke anything.

An eval without a pass bar is just content generation. Before you run a single model, write down what "good enough to ship" means, as a check you could hand to a colleague. Three families cover most workloads.

Correctness: does the answer contain the right facts, fields, or classification? This is where the exact-match and rubric work from the previous section pays off.

Format discipline: is the output parseable by the system that consumes it — valid JSON where JSON was promised, the right schema, no trailing prose? A model that is brilliant but undisciplined is expensive in a way per-token pricing never shows, because every malformed answer is a paid retry.

Refusal and edge behavior: what does the model do with the requests it should decline or cannot answer? A model that confidently invents an answer to an out-of-scope question fails differently — and worse — than one that says so.

Set the bar where your product needs it, not where it flatters a model. The bar is a product decision; the eval just enforces it.

Exact match grades itself, but rubric answers need a judge, and that is where eval projects stall: someone realizes a human has to read hundreds of outputs, and the eval quietly never happens. The practical answer is a model grader — a strong model given the rubric and asked to score each answer — with a crucial caveat: the judge is a model, with a model's failure modes. It will have blind spots, it can be charmed by fluent wrongness, and it tends to grade its own family's style generously.

The discipline that makes model grading trustworthy is the spot check. Humans grade a small slice of every run, and if the judge's scores and the humans' scores disagree beyond what you tolerate, the rubric or the judge gets fixed before anything ships. The judge exists to make grading cheap enough to run constantly, not to make human judgment unnecessary.

A second trap: letting the judge's convenience redefine correctness. If the rubric gets quietly rewritten into whatever the judge scores reliably, the eval now measures measurability. Keep the rubric owned by the product side — the same people who set the acceptance bar — and let the tooling serve it, not steer it.

Now run the candidate models against the set, and be strict about the order of operations. Quality is a gate, price is a ranking: a model either clears the acceptance bar or it does not, and the ones that do not are not "cheaper options", they are not options at all. This single discipline prevents the most common procurement error in the market — choosing a model for its price and then spending the savings on retries, guardrails, and apologies.

Among the survivors, rank by cost per completed task: the all-in cost of an accepted answer, including retries and wasted output, not the sticker rate. The framework and the traps in measuring it are in the cost-per-task post; the per-token inputs for the arithmetic are on the pricing calculator, which reads the same published rates as the model library.

Notice what this process does to the shortlist: it usually surfaces a model several tiers below the one the leaderboard suggested. That is not the eval being lenient. That is the eval doing its job.

Most teams treat evaluation as a procurement step and throw it away after the contract is signed. That is the expensive reading. The eval you just built is a regression suite for every future decision: when a new model releases, you have a same-day answer to "should we switch"; when your provider changes a price or deprecates a version, you can re-rank the survivors instead of guessing; when someone proposes a prompt change, the eval tells you whether it helped before production does.

It also converts vendor claims into testable propositions. A provider announces a new flagship; the press release says it is better; your eval says whether it is better at your work. The catalog will keep moving — that is the argument in how to read a price list for dating every comparison — and the teams that re-evaluate in hours instead of quarters are the ones whose bills keep falling as the market improves.

Keep the set small enough to run cheaply and fresh enough to match live traffic. A stale eval drifts from the workload it was built from, and a drifted eval quietly re-becomes a leaderboard: accurate about everything except your product.

If the workload is low-stakes and low-volume — an internal helper, a prototype, a tool a dozen colleagues use — a structured eval is overkill. Try the two or three obvious candidates on a handful of real prompts, pick the one that reads best, and spend your remaining effort on the prompt itself. The machinery in this post pays for itself when the workload is large enough that a wrong model choice is a recurring bill or a recurring incident.

The honest test is reversibility. If switching models later is a config change, a fast informal choice costs you little; our OpenAI-compatible API exists precisely so that decision stays cheap. If the choice is about to be baked into a contract, a budget, or a customer-facing promise, build the eval first.

Either way, write the bar down. Even a vibe check deserves a sentence about what "good" means, because in six months that sentence is the only thing standing between you and re-litigating the decision from memory.