Embeddings and Vector Search: The Hidden Cost

Most of the cost conversation about LLM systems is about the model: the tokens in, the tokens out, the per-million rate. But the systems that actually work in production have a second layer that nobody prices correctly — the retrieval layer. The embeddings that turn text into vectors, the index that stores them, and the search that finds the relevant pieces before the model ever sees the question.

This layer is not optional for most products. It is how the model knows your documents, your products, your history. And it has its own cost structure: embedding calls billed per token, index storage billed per vector, search billed per query. Three bills where most teams budgeted for one.

The layer is also where the quality lives or dies. The model can only answer from what retrieval finds; a retrieval layer that returns the wrong passages produces a confident model answering the wrong question. The cost conversation and the quality conversation are the same conversation.

A list of numbers that represents the meaning of a piece of text. The embedding model reads a sentence and produces a vector — a point in a high-dimensional space — positioned so that texts with similar meanings land near each other. "How do I reset my password" and "I forgot my password" produce nearby points; "password" and "pineapple" do not.

The search is the geometry: a new question is embedded, and the index finds the stored vectors nearest to it. That is the entire retrieval trick — meaning converted to distance, and distance computed by arithmetic.

And the embedding model is a model like any other: it has a tokenizer, a context window, and a per-token price. The cost discipline from the token post applies to it unchanged — the difference is that embedding calls are invisible, fired in bulk, and rarely counted.

Three bills, each with its own failure mode. The embedding bill: every document is embedded when it enters the system, and every query is embedded when it arrives. The document side is a one-time cost per document; the query side is a per-request cost forever. Teams that budgeted the query side and forgot the ingestion side discover the difference on the first full re-index.

The storage bill: every vector is stored, and the index grows with the corpus. The cost is per vector per month, and it compounds — a corpus that grows steadily is a storage bill that grows steadily, long after the ingestion budget was spent.

The search bill: every query costs a search, and the search cost scales with the index size and the number of results requested. The model call that follows is the expensive part, but the search is the part that runs on every request, including the ones that retrieve nothing useful.

And the hidden fourth: re-embedding. When the embedding model changes — and it will — the entire corpus must be re-embedded, and the one-time ingestion cost becomes a recurring event.

The index is the database of vectors, and it has the same lifecycle costs as any database: storage, compute for search, and the operational work of keeping it consistent with the source documents. A document changes, and the index must change with it — the update path is where retrieval systems quietly rot.

The consistency question is the expensive one: does the index update in real time, in batches, or on a schedule? Real-time updates cost the most and prevent the most embarrassing failure — the answer that cites a document that was corrected last week. Batch updates are cheaper and guarantee that failure happens sometimes.

And the index has a quality dial too: the number of results retrieved per query, the similarity threshold, the reranking step. Each dial turn changes the cost and the quality together. The retrieval layer is a system with its own tuning surface, and the tuning is part of the job.

The cheap retrieval setup — a small embedding model, a loose threshold, no reranking — retrieves fast and retrieves wrong. The expensive setup — a strong embedding model, a reranker on top, a tight threshold — retrieves right and bills accordingly. The quality of the model's answers is bounded by this choice, and the choice is a budget decision nobody made consciously.

The reranker is the clearest example: a second model that reads the retrieved candidates and orders them by relevance. It doubles the retrieval cost and dramatically improves the answers. Teams that skip it save money on retrieval and spend it on model calls that answer from the wrong passages.

The honest framing: retrieval quality is measured in the same currency as everything else on this blog — cost per completed task. A retrieval layer that finds the right passage on the first try is cheap at any price; one that finds it on the third retry is expensive at any discount.

When the corpus is small enough to fit in the context window directly. A handful of documents, a short FAQ, a product catalog that fits in the prompt — the retrieval layer is pure overhead, and the model can read everything itself. The crossover is arithmetic: when the corpus fits, retrieval is a tax.

It is also the wrong answer when the queries are exact rather than semantic. A lookup by ID, a search for an exact string, a filter on a field — these are database queries, and a vector index is an expensive way to run them. The semantic layer earns its keep on meaning, not on matching.

And it is the wrong answer when the corpus changes faster than the index can follow. Real-time data, rapidly edited documents, anything where staleness is a correctness bug — the retrieval layer becomes the source of the errors it was built to prevent. The index is only as good as its update path.