Prompt Caching and the Economics of the Static Prefix

Somewhere in your stack there is a prompt that never changes. A system message with your product's instructions. A policy document stuffed into context. A knowledge-base extract, a style guide, a schema definition. It is the same bytes, request after request, day after day.

Every one of those requests pays to process it again. Input tokens are the cheaper half of the rate card, but cheap multiplied by always is still a line item, and for context-heavy products it is often the largest single component of the bill. Teams notice their output costs and agonize over response length while the same policy document rides along ten thousand times a day, billed at full freight.

Prompt caching is the industry's answer to this, and it is genuinely good news: the repeated part of your prompt can be priced as what it is — work already done. The catch, and the reason this post exists, is that whether you collect the discount is decided by how your prompts are laid out. It is an architecture decision wearing a pricing label.

Strip the marketing away and the mechanism is intuitive. When a model processes your prompt, it builds internal state for every token — the computational residue of having read that far into the text. Building that state is most of what input billing pays for.

If your next request begins with exactly the same tokens, a caching provider can skip the rebuild: keep the state from last time, resume where the requests diverge. The repeated prefix is billed at a discounted cache-read rate on providers that offer the tier, because the work genuinely was not repeated.

Two properties fall out of the mechanism and both matter more than the discount. First, matching starts at the very first token: caches hit on prefixes, so the identical content must come at the beginning of the prompt, not merely somewhere in it. Second, exactness is unforgiving — one changed byte near the start can invalidate everything after it. A cache-friendly prompt is not a prompt that mostly repeats; it is a prompt whose stable content is first and whose volatile content is last.

Most prompts are written for the model's benefit, in whatever order reads naturally. The prefix rule adds a second reader: the cache. The layout that satisfies both is stable-to-volatile — everything that is the same across requests first, everything that varies last.

Concretely: system instructions and standing documents at the head, then any session-stable context, then the variable tail — retrieved passages, the user's actual question, per-request metadata. If your template currently injects anything volatile near the top, you are paying full input price on everything below it, every time.

This occasionally conflicts with a prompt-ordering habit you have for quality reasons, and quality wins — a cache discount is worth less than a correct answer. But in our experience the conflict is rarer than teams assume. Most prompt orderings were never chosen deliberately at all; they are just where the template happened to concatenate things.

The usual suspects, in order of how often we see them:

• **Timestamps and dates near the head.** "Today is..." in the system prompt makes every request unique. If the model needs the date, put it at the tail.
• **Per-user identifiers up front.** Personalization is legitimate; placing it at token one is a billing decision you did not mean to make. User-specific context belongs at the end.
• **Non-deterministic ordering.** Few-shot examples or retrieved documents assembled in random or ranking-dependent order produce a different prefix per request. Sort them canonically.
• **Helpful middleware.** Request IDs, trace headers, and session metadata injected at the top of the prompt by a framework layer nobody has read since launch.
• **Silent template drift.** Two code paths that build the same prompt with one cosmetic difference — a trailing newline, a reordered key — each maintain their own cold cache.

None of these are bugs. All of them are money.

Before and after, stripped to structure. Most teams start here — natural to write, hostile to the cache:

[request id + timestamp]          <- volatile, at the worst position
[user name and plan]              <- volatile
[system instructions]             <- stable, but unreachable
[policy document]                 <- stable, but unreachable
[user question]                   <- volatile

Every request differs at the very first token, so nothing downstream is ever reused. The stable two-thirds of the prompt pays full input price forever. The same prompt, reordered for the cache:

[system instructions]             <- stable, first
[policy document]                 <- stable, second
[user name and plan]              <- volatile, near the tail
[user question]                   <- volatile, last
[request id, if the model needs it] <- usually it does not

Same words, same model, same answer quality — but the first two blocks are now identical across requests, which is the entire game. Note that nothing was deleted and nothing was summarized; the only change is respecting the prefix rule. That is why prompt layout is a billing decision: the cache does not care what your prompt says, only what stays still at the front of it.

If your prompts are short, there is barely a prefix to cache, and the discount applies to a rounding error. If every request is genuinely novel — a tool that processes entirely different documents with no shared context — the workload has no repetition to exploit, and the stuffing-vs-retrieval question becomes the more useful frame (see long context vs RAG). And if your traffic is sparse, the cache may simply expire between requests: cached state lives as long as the provider keeps it warm, and a workload that calls in bursts separated by long silences will find the cache cold each time.

There is also a trap in the other direction: reorganizing a prompt to be cache-shaped can cost more than it saves if the reorganization makes the prompt longer or the answers worse. Cache discounts are a tailwind for a well-built prompt, not a reason to build a strange one.

The honest first step is measurement. Usage responses report how much of your input was served from cache on providers that expose it — watch that share before and after any layout change, on real traffic, before declaring victory.

Prompt caching is one lever among several, and it helps to place it honestly. Deleting context the model never needed is strictly better than caching it — a cached token is discounted, a deleted token is free. Routing requests to the smallest adequate model multiplies the whole bill down, cached rows included. Caching is what you do with the static context that survives both audits: the instructions, documents, and schemas that genuinely earn their place in every request.

Seen that way, the order of operations is delete, then route, then cache. Teams that reach for caching first occasionally end up efficiently reprocessing context they should never have sent.

For the deletion pass, the price-list arithmetic in How to Read an LLM Price List tells you what the input row is worth on your volumes; our published rates are on the pricing page when you want real numbers to run it with.

When the static prefix is small or the volume is modest, the entire exercise optimizes a line item nobody will notice. Check the share of your bill that is repeated input before scheduling the work; the answer is frequently "not enough to matter yet".

When quality is unsettled, freeze the prompt for correctness before you freeze it for caching. A prompt still being tuned weekly will never hold a cache anyway, and contorting it prematurely buys a discount on the wrong artifact.

And when the prompt is shared because it has grown into an unowned dumping ground — every team appending instructions nobody removes — the problem is governance, not caching. Split it, assign it an owner, and let the cache benefit fall out of the cleanup. It usually does.