Why Hard-Coding One Model Is Now Your Biggest AI Cost Line

Somewhere in your codebase there is a string that looks like model="...", and it is setting the price of everything your product does. Not because the model is wrong — it is probably a good model — but because the choice was made once, early, and has never been revisited while the workload grew up around it.

This is the routing problem, and it has quietly become the largest controllable line in most AI budgets. Not the model prices themselves. Not the infrastructure. The fact that a request that needed a cheap classification and a request that needed multi-step reasoning are paying the same rate, because one string says so.

This post is about the category, not about our product. We build one answer to it — BiOS Adaptive — and we will show it alongside the others, because a post that only argues for its own product is a pitch, and you can get those anywhere.

Three things, and only one of them is visible on the invoice.

The visible one: every request bills at that model's rate. A frontier model charges frontier prices for work that never needed it. A budget model saves money on easy work and then quietly fails on hard work.

The second: the model's capability ceiling becomes your product's ceiling. Features that the pinned model cannot do well — long documents, tool use, vision — get designed away in planning meetings rather than evaluated, because the model id is treated as fixed infrastructure.

The third is the one nobody notices: the pinned model freezes in time. The catalog around it moves constantly. New models arrive, prices fall, and the gap between what you chose and what you would choose today widens every month you do not look.

Because defaulting to the strongest model is the safe choice, and safe choices are how budgets die. No engineer ever got fired for routing a support ticket classifier through a frontier model, and the per-request cost looks trivial in isolation. It stops looking trivial when you multiply it by every request, every day, forever.

Real products are mixed workloads. Trivial prompts — intent classification, entity extraction, yes-or-no guardrails — sit beside genuinely hard problems that justify strong models. When everything rides one model, either the easy work is over-served or the hard work is under-served. Usually it is the former, because the visible failure mode of under-serving is angry users, while the failure mode of over-serving is a finance conversation months later.

The uncomfortable part: teams usually know this. What they lack is not awareness but a mechanism. Something has to decide, per request, which tier of model the request deserves. That something is the routing layer, and building it is where most teams stall.

Strip away the vocabulary and routing is a classifier in front of your catalog: look at the request, predict what it needs, send it there. The implementations differ enormously in sophistication.

Rule-based routing is the simplest: keyword checks, length thresholds, explicit task types passed by your own code. It works, it is transparent, and it encodes whatever you already know about your traffic. Its ceiling is that your rules are only as good as your taxonomy, and traffic drifts out from under taxonomies.

Cascade routing sends everything to a cheap model first and escalates on low confidence or failed validation. It is cheap to build and surprisingly effective, but it pays a latency tax on every escalated request (the mechanics are in the latency post) and can thrash when the cheap model is confidently wrong.

Learned routing uses a model or a trained classifier to predict difficulty and pick the target before any token is generated. This is the approach that scales to a large catalog, and it is the approach our own adaptive endpoint takes. It is also the hardest to build well yourself, because the predictor needs to keep pace with every model in the catalog as they all change.

In the over-serving direction, the cost is direct and measurable: you pay frontier rates for commodity work. It never shows up as a line item called waste. It shows up as a bill that feels structural, because it is baked into architecture.

In the under-serving direction, the cost hides. A budget model that mangles a hard request does not just deliver a bad answer — it triggers retries, human escalations, and downstream corrections, each of which costs more than the saving. Teams then compensate by bolting on validation layers and retry loops, and end up paying frontier-grade complexity on top of budget-model pricing.

The worst outcome is the one that looks like prudence: pinning a mid-tier model for everything. It over-serves the easy half of the workload and under-serves the hard half simultaneously, which is how a reasonable-looking model choice ends up expensive and mediocre at the same time.

You can build it. If your traffic splits cleanly into a few known task types, rule-based routing is a week of work and entirely respectable. Own the rules, own the fallbacks, revisit them quarterly. For many products this is the right answer.

You can assemble it. Open-source gateways and proxy layers give you multi-provider routing primitives — model fallbacks, load balancing, per-model keys — and you supply the difficulty logic on top. More moving parts than building rules, fewer than building a learned router from scratch.

Or you can rent the decision. An adaptive endpoint like BiOS Adaptive takes the whole problem: one model id, and the platform picks the serving model per request, bills you at the rate of whatever actually answered, and fails over internally instead of returning errors. The trade is transparency — you are trusting the platform's judgment per request — against the engineering you do not have to do. Whether that trade makes sense depends on how mixed your workload is and how much of your team's time you want to spend on model operations. The model library shows the catalog such an endpoint draws from.

Routing and fine-tuning answer different questions, and the interesting architecture is the one that uses both. Fine-tuning changes what a model is; routing changes which model a request meets. A tuned model that embodies your domain is exactly the kind of specialist that deserves its own tier in a routing plan — not a reason to abandon the plan.

The trap is treating a fine-tune as a reason to pin everything to it. Your tuned model is presumably excellent at the task class you trained it for. The requests outside that class — the guardrail check, the off-topic question, the formatting job — gain nothing from it and still bill at whatever serving tier it sits on. Specialists are for special work.

The composition we see work well: tune where you have data and a durable task, route where the workload is mixed, and revisit the boundary whenever the catalog or your traffic shifts. Neither mechanism replaces the other.

If your product does exactly one thing, there is nothing to route. A single-task pipeline gains nothing from per-request model selection, and a pinned model with a well-tuned prompt will beat any general mechanism on both cost and predictability.

If you need a frozen model for regulatory or evaluation reasons — an audit trail that must name the exact weights that produced every output — adaptive routing is genuinely wrong for you. Pin, log, and do not let anyone make it clever.

And if your latency budget is tight enough that the routing decision itself matters, measure before adopting anything. Routing adds a decision step. It is small, but small is not zero, and you should know what it costs in your path rather than taking our word for it.

The useful property of OpenAI-compatible APIs is that the experiment is a two-line change. Point the client at a new base URL and change the model id; nothing else in the code moves. Against an adaptive endpoint it looks like this:

from openai import OpenAI

client = OpenAI( base_url="https://api.runbios.ai/v1", api_key="sk-bios-...", )

resp = client.chat.completions.create( model="bios-adaptive", messages=[{"role": "user", "content": "Classify this ticket"}], ) ```

Run it in shadow mode first: duplicate a slice of production traffic, compare outputs on the tasks you care about, and look at what the blend actually cost. The API overview covers authentication and the rest of the surface. The experiment costs an afternoon and tells you more than any comparison page — including ours.