Streaming: Why First-Token Time Beats Total Time

The latency post measured what the clock says. This post is about what the user feels, and the two disagree in a specific, exploitable way. A user staring at a blank screen experiences every second as an eternity; a user watching words appear experiences the same total time as progress. The clock is identical. The experience is not.

Streaming is the difference: instead of waiting for the complete answer and rendering it at once, the model emits tokens as they are produced and the interface renders them as they arrive. The total time does not change. The perceived time collapses.

This is not a cosmetic trick. Perceived latency is the latency that churns users, and the streaming decision is one of the cheapest product improvements available — it costs no tokens, no model changes, and no infrastructure.

First-token time is the gap between the request leaving and the first piece of the answer arriving. It is dominated by the things that happen before generation starts: the queue, the prefill of your prompt, the round trip. Total time adds the generation itself — the long tail of tokens streaming out.

The perception asymmetry: a user will forgive a long generation if it started quickly, and will not forgive a long silence even if the total is short. First-token time is the silence; generation is the progress. The latency post's three numbers — first token, per-token rate, total — map onto the experience in exactly this order of importance.

Which is why the instrumentation advice from that post matters here: you cannot improve perceived latency you have not measured, and the first-token distribution is the measurement that predicts the feeling.

Three things, and none of them is the model. The transport: instead of one response at the end, the API delivers a stream of chunks, each carrying a few tokens, and the client renders them incrementally. The rendering: the interface must be built to append — a chat bubble that grows, a table that fills — rather than to swap in a finished block. And the failure handling: a stream can end early, which is the truncation case the resilience post covers, and the client must know the difference between a finished answer and a dropped connection.

Streaming also changes the cancellation economics. A user who stops reading can stop the generation — the request is cancelled, the remaining tokens are never produced, and the bill stops where the attention stopped. Non-streaming systems pay for the whole answer whether anyone read it or not.

None of this requires exotic engineering. The API surface is standard, the client work is a rendering loop, and the payoff is the largest perceived-latency win available for the cost.

When the consumer is not a human. A pipeline that parses the answer needs the whole answer before it can begin — streaming to a parser is just delivery in pieces, and the pieces add complexity without adding value. Batch workloads, eval runs, anything whose deadline is measured in hours: the batch post's rule applies, and nobody is watching the spinner.

Streaming also hurts when the interface is not built for it. A UI that re-renders the whole block on every chunk flickers and stutters; a stream rendered badly is worse than no stream at all. The rendering loop is the product, not the transport.

And it hurts when the answer is short. A one-line response arrives in a single chunk either way, and the streaming machinery is pure overhead. The decision is per-surface, not per-product.

The interface work is small and specific. Render the first chunk the moment it arrives — do not wait for a sentence boundary or a minimum length; the first token is the psychological event. Show the thinking: a cursor, a typing indicator, the model's name — anything that converts waiting into watching. And keep the layout stable: reserve the space the answer will occupy, so the page does not jump as the text grows.

On the measurement side, instrument the two numbers that matter to the feeling: time to first visible token, and the gap between chunks. A stream that stalls for seconds between chunks feels broken even when the average is fine — the tail discipline from the latency post applies to chunk gaps too.

And respect the reader's time: the cancellation path is a feature. A user who can stop a wrong answer early is a user who stays for the next question.

You cannot, directly — perception is not a clock. What you can measure is the proxy that predicts it: first-token time at the percentiles that matter, chunk-gap stalls, and abandonment. Abandonment is the honest metric: how often users leave before the answer finishes, plotted against first-token time. When the curve bends, you have found the number your users feel.

The instrumentation is the same three timestamps from the latency post, plus one: the moment the first chunk was rendered, not just received. The gap between received and rendered is your own code, and it is usually the cheapest milliseconds available.

Measure, then stream, then measure again. The order matters — streaming is the fix for a specific feeling, and the measurement tells you whether the feeling moved.