Open vs Closed Models in Production: Cost per Completed Task, Not Cost per Token

Every model comparison on the internet ranks by the same number: price per token. It is an easy number to publish and an easy number to sort, and it answers a question nobody actually has. You do not buy tokens. You buy completed tasks — resolved tickets, extracted fields, drafted replies, working code. The token is just the unit the bill is printed in.

Between the token price and the task price stands everything a model can do wrong: ramble, refuse, emit broken JSON, fail a validation and get retried, or produce an answer a human has to redo. Each of those consumes tokens you paid for and contributes nothing to a completed task. Two models can sit far apart on a per-token table and land in the opposite order once you price the finished job.

This post is a framework, not a leaderboard. We are not going to publish a table of our catalog ranked by cost per task, because we have not run that benchmark, and a ranking without a benchmark behind it is exactly the kind of content this article argues against. What we can give you is the method, so the number you act on is measured on your workload instead of invented on ours — the how-to is in the eval-before-purchase post.

Verbosity is the first leak. Models differ enormously in how many tokens they spend to say the same thing — reasoning traces, restatements, hedging, courtesy padding. Output tokens are usually the expensive half of any rate card, so a model that talks more bills more, even when the extra words add nothing. A cheap-per-token model with a long-winded style can cost more per answer than a terse premium one.

Retries are the second leak. If a model fails your format check or your quality bar on some share of requests, you pay for the failures and then pay again for the recovery. The effective price of a model is its token price divided by its success rate on your tasks — and the success rate is the part no pricing page can tell you.

The third leak is subtle: the cheapest model that cannot do the task at all has an infinite cost per completed task. Teams discover this when they route work down-market to save money and end up building validation, repair prompts, and human escalation paths around the gap. The engineering around a too-weak model is part of its price.

Define it precisely and it stops being a slogan. Take a fixed set of real tasks from your product. Run each candidate model over the whole set. A task counts as complete only when its output passes your acceptance check — schema-valid, correct, shippable, whatever your bar is. The model's cost per completed task is everything you were billed across the run — first attempts, retries, repair prompts, all of it — divided by the number of tasks that passed.

Written as a loop, the measurement is deliberately unglamorous:

for task in tasks:                    # your real prompts, frozen
    attempts = 0
    cost = 0
    while True:
        attempts += 1
        result = call_model(model, task)
        cost += result.billed_cost    # every attempt counts
        if passes_acceptance(result): # your bar, not a benchmark's
            break
        if attempts > max_attempts:   # a failure is a cost too
            break
    record(model, task, cost, passed)

The details that make it honest are all in the discipline: the task set is frozen before you start, the acceptance check is written before you see any outputs, and failures count at full cost. Change any of those after looking at results and you have an advertisement, not a measurement.

Build the task set from production, not from a benchmark suite. Sample real requests across the actual mix of work your product does — including the boring majority, because that is where the money is. Weighting matters: a model that shines on your hardest showcase task but drowns on the everyday volume is the wrong buy.

Freeze everything before the first call: the tasks, the prompts, the acceptance checks, the retry policy. Then run every candidate against the identical rig. The temptation to tune a prompt mid-run for the model you expected to win is strong, and it converts the exercise from measurement into advocacy. If a prompt needs to change, the run restarts.

Keep the raw outputs, not just the scores. When the ranking surprises you — and it will — the artifacts are how you find out whether you measured the models or a bug in your harness.

The table inverts more often than anyone expects. Terse, well-behaved models climb; eloquent ones sink. Models that follow output schemas reliably beat smarter models that freelance on format, because a failed parse is a retry, and a retry is a full-price second attempt.

The open-versus-closed question changes shape too. Per-token tables make it a simple price ordering. Per-task measurement makes it a question about task classes: open models are strong, and often excellent, on the structured work that makes up most production traffic — classification, extraction, transformation, grounded question-answering. The gap that remains tends to concentrate in open-ended, ambiguous, or genuinely novel reasoning. Which share of your workload that is decides the answer, and you cannot know the share until you measure it.

The catalog you would run this against — open frontier models alongside the closed ones, all behind the same API shape — is on our model library, and the per-token side of the math is already done for you on the pricing calculator. The task-cost side is the part only you can supply.

Cost per completed task is one axis, and treating it as the only axis is how teams buy cheap chaos. The measurement only means anything above a quality floor you set in advance: the acceptance check defines "complete", and if the bar is set where your users would be disappointed, you have priced work you would never ship. Set the floor first, then minimize cost above it — never the other way round.

Latency belongs in the same frame. A model that is cheap per task but slow enough to hurt the experience is not cheap; it is subsidized by your users' patience. Interactive products should bound acceptable latency before looking at the ranking, and batch pipelines should notice that their constraint runs the other way — throughput per dollar, not milliseconds per request.

The honest summary is three numbers per model, not one: what a completed task costs, how often it clears the quality bar, and how long it takes. Anything simpler is a lobby poster.

When the task is genuinely novel. Frontier closed models earn their rate on work with no template: ambiguous specifications, long-horizon reasoning, problems where the cost of a wrong answer dwarfs the cost of the tokens. If a failure means a lost customer or a bad headline, the premium model is the cheap option.

When the volume is small. If a task runs a handful of times a day, the difference between a good model and the best model is pocket change, and the measurement exercise costs more than it will ever save. Spend the rigor where the volume is.

And when your acceptance check cannot be automated, per-task costing still works — it just needs a human in the loop, which makes it slower and more expensive to run. That is an argument for measuring a smaller, higher-stakes task set, not for skipping the measurement.

Everything you need is unglamorous: a frozen sample of your real traffic, an acceptance check you trust, and API access to the candidates. Because every model on Run BiOS speaks the same OpenAI-compatible API, one harness can sweep the whole catalog by changing a model id — the API overview covers the setup, and the loop above is genuinely all there is to it.

If you would rather not hand-model the blend at all, BiOS Adaptive exists for exactly the workload this article describes: mixed traffic where the right price-quality trade-off differs per request. But whether you use it or not, measure first. The per-token price is a fact about a model. The cost per completed task is a fact about your product, and it is the only one of the two your CFO is actually paying.