Adaptive inference: matching the model to the work

The observation this rests on is mundane enough that it is usually skipped over. The questions an organisation asks of a model are not equally hard.

Classifying an invoice, extracting a date, routing a ticket, drafting a standard reply and summarising a short document are not the same kind of task as reasoning through a contradictory contract or debugging an unfamiliar system. In a typical enterprise workload the easy cases dominate by volume and the hard cases dominate by consequence.

Almost every deployment answers all of them with one model chosen for the hardest case, because choosing per request is work and choosing once is not. That is a rational decision for a team with other problems. It is also, at volume, the largest recurring inefficiency in most AI budgets.

The intuitive expectation was that falling prices would make allocation irrelevant. The opposite happened.

Over a single year the blended price of a million tokens fell by roughly two thirds while average enterprise AI budgets rose about sixfold[1]. Cheaper units did not reduce spend, they removed the friction that had been limiting volume. Spending on the model and platform layer is growing at more than sixty per cent a year[4].

This is the ordinary history of any commoditised input. The value does not disappear, it relocates to allocation and to specialisation. Adaptive inference is the allocation half of that answer; training a model on your own work is the specialisation half. They are complementary rather than competing, which is the point of section four.

Allocation is easy to do and hard to do safely, and the difficulty is entirely in knowing whether you broke something.

Sending less demanding work to a smaller model reduces cost immediately and visibly. Whether it reduced quality is a question that does not answer itself, because the cases where it did are rare by construction and distributed across a long tail. A naive system optimises the number it can see and degrades the one it cannot.

The literature has converged on two workable framings. Cascading treats the problem as escalation: try cheaply, detect insufficiency, retry expensively, and pay the overhead on the minority of cases[2]. Learned allocation treats it as prediction, using preference signal to estimate in advance whether the cheaper answer would be judged as good[3]. Both depend on a quality signal that is honest, and getting that signal is the actual work.

Our position is that the signal has to come from the same place the traffic does. Held-out benchmarks measure a distribution that is not yours. The only evaluation that settles the question for a given organisation is drawn from that organisation's own requests and its own judgements about which answers were acceptable, which is the same material the continuous-learning loop consumes.

Treated separately, these are two cost levers. Treated together, they are one system, and each makes the other better.

A system that allocates work across models produces, as a by-product, a labelled record of which kinds of work needed depth and which did not. That record is exactly the curation signal a training loop wants. Conversely, every time specialisation succeeds, a class of work that previously required an expensive model stops requiring one, which changes the allocation the router should make.

The end state is a system where the expensive general model is used for genuinely novel problems, an owned specialised model handles the work the organisation does repeatedly, and the boundary between them moves on its own as the specialised model improves. Neither half reaches that alone.

It is worth being plain that this is a direction of travel rather than a finished result. The industry's record on enterprise AI outcomes is poor[5], and a claim to have solved allocation would deserve the scepticism it got. What we will defend is the framing: allocation and specialisation are the two remaining levers once capability is cheap, and a platform that owns both can move them together.