The Cheapest Request Is the One That Can Wait

When teams optimize an inference bill, they reach for two levers: which model, and which provider. There is a third lever that gets asked about far less: when. Not when in the cron sense — when in the deadline sense. How long can this specific request wait before its answer stops being useful?

That question is a pricing question, because urgency is what the real-time rate buys. A request answered in a heartbeat and a request answered by morning consume the same computation, but they are different products, and they deserve different price tiers. Providers know this: deferrable work fills idle capacity that would otherwise earn nothing, so work that can wait is worth a discount.

Most workloads were never asked the deadline question. They were built interactively because that was the default, and entire categories of deferrable traffic — reports, evaluations, backfills, enrichment — have been paying the urgency premium ever since. This post is about finding them.

Real-time traffic is any request where a human or a downstream system is actively waiting. Chat interfaces, copilots, interactive search, agents mid-loop. The latency budget is part of the product here — a correct answer that arrives after the user has tabbed away is not a correct answer, it is a contribution to a log.

This traffic earns its price tier. What you are paying for is priority: the request jumps the queue, capacity is held ready for it, and the tail latency is engineered down because somebody's attention is on the line. Trying to save money by degrading this half is borrowing against churn.

The one optimization that belongs here is model choice, not scheduling: the smallest model that clears the quality bar, per request. Right-sizing a chat request saves money without touching its urgency. After that, leave the real-time path alone and go find the other half of your traffic.

The batch half is everything whose deadline is measured in hours: the nightly report nobody opens before nine, the evaluation suite that gates tomorrow's deploy, the document backlog being enriched, the embeddings being rebuilt. This work is large, predictable, and patient — three properties providers prize, because patient work can be packed into the gaps that interactive traffic leaves.

That packing is why batch tiers exist on price lists, and it is the same mechanism that makes below-list pricing possible in the first place — we walked through the utilization economics in how we price GLM 5.2 below list. Aggregation and batching are not marketing words; they are the physical reason deferred work is cheaper to serve, and the discount is the saving being shared.

The quiet scandal is how much batch-shaped traffic still rides the real-time tier. It happens innocently: the pipeline was prototyped against the interactive endpoint, nobody revisited the decision, and the cron job has been jumping the queue ever since.

Between "user staring at a spinner" and "read at nine tomorrow" sits a band of work that is easy to misclassify: the notification that should arrive within minutes, the review queue a human works through over a morning, the pipeline stage whose downstream step runs hourly. None of these need heartbeat latency, and none of them can wait all night.

The useful discipline is to give this band its own service-level statement instead of letting it inherit the real-time default. "Within minutes" is a designable target: a short queue, a modest polling interval, a worker that drains promptly. It costs a fraction of the urgency premium precisely because you have replaced guaranteed immediacy with an honest, bounded delay.

Human-in-the-loop workflows are the biggest residents of this band. If the output lands in a queue a reviewer checks between other tasks, the model's speed was never the constraint — the reviewer's afternoon was. Pricing such traffic as real-time is paying for a spinner nobody watches.

The test is one question per workload: who or what is waiting, and what does a delay cost? If a user is staring at a spinner, the answer is real-time and the conversation ends. If the output lands somewhere it will be consumed hours later — a dashboard refreshed nightly, a queue processed in the morning, a record nobody reads until audit season — the work is deferrable whether or not it is currently deferred.

The interesting cases sit in between. Some products have asynchronous-but-soon requirements: a notification that should arrive in minutes, a pipeline stage whose downstream step starts on the hour. Treat the deadline as a spectrum, and place each workload at the latest point on it that the product honestly tolerates. Most teams discover their traffic sorts into a small urgent slice and a large patient mass, which is exactly the split the pricing tiers were built for.

Resist classifying by importance. Important and urgent are different axes: a compliance report can be the most important job of the night and still have all night to run.

Less dramatic than it sounds: the same API, the same models, two paths through your own system. Interactive requests go straight to the endpoint as they do today. Deferrable work goes through a queue — a table, a topic, whatever you already run — drained by a worker that submits on a schedule and writes results back where the morning expects them.

The queue earns its keep beyond pricing: it absorbs retries gracefully, smooths your rate-limit footprint (the four absorption patterns are in the rate-limit post), and turns "the provider had a bad hour in the night" from a page into a non-event. Deferred work is naturally resilient work, because the deadline has room in it.

One practical note on capacity shape: if your patient mass grows large and steady enough, the question stops being batch-versus-real-time and becomes serverless-versus-dedicated — that crossover has its own arithmetic, which we laid out in the break-even post. The deployments docs cover both paths. Price your actual split with the calculator before committing to either.

The first trap is misclassification in the optimistic direction: deciding something is deferrable when it is not. The test is the consumer, not the producer — if any path from that output leads quickly back to a waiting human, it is interactive work wearing a batch costume, and the saving will be paid back in abandoned sessions.

The second is misclassification in the pessimistic direction, which is more common and more expensive: interactive defaults applied to work nobody waits on. Every internal tool, every back-office pipeline, every "temporary" script that became load-bearing tends to inherit the interactive tier from the day it was prototyped.

The third is operational: batch work fails silently unless you make it loud. A failed interactive request complains immediately — someone retries, someone notices. A failed batch job is discovered in the morning, sometimes. Deferred pipelines need their own alerting and their own retry discipline, because the whole point of deferring is that nobody is watching at the time.

When the product is genuinely, wholly interactive, there is no batch half to find — a chat product with no reporting surface should not invent one. The lever is model choice, and this post is not for you.

When you are still searching for product quality, batch thinking is premature twice over. During development you want feedback loops short, not cheap — a slow, inexpensive experiment cycle teaches you less per week than a fast one, and learning speed is the budget that matters before product-market fit. Optimize the bill after the workload is stable enough to have a shape worth pricing.

And when the patient mass is small, the engineering of a queue, a worker, and its alerting can exceed the discount for a long time. Run the arithmetic on your real volumes first; deferral is a lever, not a religion. The pricing calculator will tell you in minutes whether your split is worth building.