Latency Is a Feature You Pay For

Ask a room what "the latency" of a model is and you will get one number. The system does not have one number to give. A request's timeline has three distinct segments, and they have different causes, different costs, and different fixes.

Time to first token is everything before the answer starts: the queue in front of you, the prompt being read and processed, the scheduling of your request onto hardware. It is dominated by input length and provider load, and it is what a user stares at before anything moves.

Per-token speed is the rhythm of the answer once it starts — how fast successive tokens appear. It is dominated by the model's size and the hardware serving it, and it sets how long a long answer takes to finish.

Total time is the sum a wall clock sees, and the only one your logs record by default. The practical point: "make it faster" is not a plan until you know which of the three is slow. A request can have a snappy first token and a glacial finish, or sit in a queue and then stream instantly. They look identical in a total-time average and are opposite problems.

The asymmetry that surprises everyone: reading your prompt is parallel, writing the answer is not. The model processes the entire input at once — that is the prefill — but produces output one token at a time, each token depending on every token before it. There is no shortcut and no parallelism to buy: a long answer takes its tokens times the per-token speed, full stop.

Two consequences follow. First, output length is the strongest latency lever you own. A prompt that invites the model to ramble costs wall-clock time exactly as surely as it costs tokens — the two are the same resource. Tighter instructions about answer shape buy both. Second, this is why verbose models feel slow even on fast hardware: the per-token rate can be excellent and the total time still poor if the model uses four hundred tokens where forty would do.

When you budget latency for a feature, start from the answer length the product actually needs, and design prompts and output caps around it before reaching for anything more exotic.

Streaming changes when the user sees the tokens, not when the model produces them. The total generation time is identical either way; what changes is that the user watches the answer arrive instead of staring at a spinner for its full duration. Perceived latency collapses, actual latency does not move.

That is not a trick, it is the correct engineering trade: human patience is spent mostly on silence, not on duration. An answer that begins immediately and takes a while reads as fast; the same answer delivered all at once after the same total time reads as broken. Any interface with a human on the other end should stream.

The machine-facing case is the opposite. A pipeline that consumes the full answer gains nothing from streaming and pays a small complexity cost for it — partial responses, reconnection logic, incomplete JSON. Stream for humans, buffer for machines, and do not let the two defaults leak into each other's lanes.

Model size is the big one: smaller models decode faster on the same hardware, which is the latency half of the right-sizing argument we made in the hidden cost of hard-coding one model. If a request's answer does not need frontier strength, it does not need frontier decode time either.

Output length, from the previous section, is the second knob, and the cheapest to turn.

The third is your position in the provider's day. Shared capacity has rush hours; a request queued behind a burst waits before its first token regardless of the model you chose. If your workload includes anything deferrable, moving it off the peak is free latency for the traffic that stays — the scheduling version of the argument in the cheapest request is the one that can wait.

The fourth knob is geography: the round trip between you and the endpoint is part of first-token time, and no provider can beat the speed of light for you. If your users and your inference are on different continents, some of your "model latency" is actually a plane ticket.

Averages hide the experience that churns users. A feature whose median request feels instant but whose one-in-a-hundred request hangs for ages is, for one percent of your sessions, a broken product — and those sessions are disproportionately the ones with long prompts and hard questions, which is to say your most engaged users.

So when you instrument, measure the distribution, not the mean: the median for the typical feel, and the high percentiles for the worst case your product tolerates. Then set the budget at the tail. "The median looks fine" is not a latency story.

The tail also tells you where to spend. A tail dominated by queueing points at capacity and scheduling; a tail dominated by long generations points at prompt and output design; a tail that spikes with specific request shapes points at those shapes. Same symptom, three different purchases — the measurement is what stops you from buying the wrong fix.

Instrumentation here is cheaper than most teams expect, because the three numbers fall out of timestamps you can already capture. Log the moment the request left, the moment the first token arrived, and the moment the response closed, for every call, tagged with the model and the prompt and completion sizes. Time to first token, decode duration, and total time come out of those three stamps, segmented by whatever dimension you suspect: model, feature, prompt length, hour of day.

A week of that log usually settles the argument that started the investigation. The first-token curve by hour tells you whether you are queueing behind other people's peaks. Decode time versus completion length tells you whether the model or the verbosity is at fault. And the tail, sliced by request shape, tells you which feature owns the bad experiences.

One measurement warning: do not benchmark from a laptop on one prompt and call it the model's speed. Cold calls, single samples, and synthetic prompts produce numbers about your test, not about the service. Latency is a distribution over real traffic; measure it there or not at all.

If nobody waits on the output, latency is a cost center pretending to be a metric. Batch pipelines, nightly jobs, evaluation suites — their deadline is measured in hours, and paying for their speed is buying a spinner nobody watches, as we argued in the batch post.

It is also the wrong focus when the real problem is correctness. A fast wrong answer is not a latency success, and teams routinely polish response times while the acceptance rate quietly bleeds. Speed is a multiplier on value, not a substitute for it.

And it is premature before you have the three numbers. Measure first token, per-token rate, and total separately; find the segment that is actually slow; then spend. Latency work done before that measurement is tourism.