Semantic Caching: When the Answer Already Exists

The word "caching" appears in two LLM contexts that have almost nothing in common, and conflating them costs money. Prompt caching, covered in its own post, stores the static prefix of your prompt so repeated requests do not re-pay for the same instructions. It is exact: the same bytes, recognized, billed once.

Semantic caching stores answers: when a new question arrives that means the same thing as a question already answered, the stored answer is returned instead of calling the model at all. It is approximate: not the same bytes, but the same meaning.

One saves the prefix; the other saves the whole call. The first is a billing optimization; the second is an architectural one — and like most architectural optimizations, it is brilliant where it fits and dangerous where it does not.

Three things: the question, in a form that can be compared — usually an embedding, a vector that represents meaning rather than text; the answer, exactly as it was served; and the metadata that decides whether the answer is still trustworthy — when it was generated, what it depended on, and how similar a new question must be to reuse it.

The matching is the interesting part. A new question is embedded, compared against stored questions, and if the similarity clears a threshold, the stored answer is served. "How do I reset my password" and "I forgot my password, what do I do" are different strings and the same question — the cache is the thing that knows.

The threshold is the entire product decision. Set it loose and the cache serves wrong answers confidently; set it tight and the cache never fires. Everything else in this post is the engineering around that one dial.

When your traffic repeats itself semantically. Support products where the same questions arrive in different words, documentation assistants, FAQ-style workloads — these have hit rates that make the cache the cheapest infrastructure in the stack, because the marginal cost of a hit is a vector comparison instead of a model call.

The arithmetic is the cost-per-task framework with a new term: the all-in cost of an accepted answer now includes the cache's miss rate. A cache that hits often enough to pay for its own misses is free latency and free tokens; the crossover is measurable, and the measurement is the only way to know.

And the latency win is real even when the cost win is small: a cache hit returns in milliseconds, which is the perceived-latency argument from the latency post, delivered by architecture instead of rendering.

The failure mode that keeps honest engineers awake: the cache serves an answer that is almost right and subtly wrong, and the user cannot tell. A threshold set too loose, a question that is similar in words and different in meaning, an answer that was correct last month and is not now — the cache converts these into confident errors, and confident errors are the most expensive kind.

The defenses are the same ones the eval post prescribes for judges: sample the hits. Log what the cache served, review a slice of it, and measure how often a human would have wanted the model called instead. A cache is a model with one training example per question; it deserves the same scrutiny.

And the escape hatch: for the questions where wrongness is expensive, the threshold tightens or the cache turns off. Caching is per-surface, not per-platform — the support bot may cache aggressively while the medical assistant does not cache at all.

Every cache demo shows the hit. None shows the day the underlying truth changes — the price update, the policy change, the new model release — and the cache keeps serving the old answer. Invalidation is the discipline of knowing what an answer depends on and expiring it when the dependency moves.

The cheap version is time: answers expire after a window, and the window is chosen per surface. The honest version is dependency: when the document an answer was built from changes, the answer dies with it. The honest version is more work, and it is the difference between a cache and a liability.

And the versioning post's lesson applies: the model itself is a dependency. An answer generated by last quarter's model is stale the day the model changes, and a cache that does not know which model produced its entries cannot know when they expired.

When the questions do not repeat. A creative drafting tool, a coding assistant on novel problems, anything where every request is genuinely new — the cache is a database of misses, and the embedding infrastructure is pure overhead.

It is also wrong when the cost of a wrong answer is high and the cost of a model call is low. The cache saves tokens; it risks trust. Where trust is the product, the tokens are the cheaper currency.

And it is premature before the traffic exists. A cache sized for questions you have not seen yet is speculation; the hit-rate data comes first, the cache second. Measure the repetition, then buy the infrastructure — the same order as every other purchase on this blog.