LLM Inference Engineering
The engineering side of serving models in production: model routing, prompt caching, API migration, and vendor evaluation. Written for the people who own the endpoint when it pages.
Structured Outputs: JSON You Can Actually Parse
The most common LLM integration failure is the answer that will not parse. JSON mode, schema enforcement, and the validation gate that makes output trustworthy.
Prompt Injection: The Threat Nobody Budgeted For
Prompt injection needs no exploit — just a sentence in the input. The threat model, the defense layers, and why the tool list is the blast radius.
What Is a Token, Anyway?
Tokens are not words, and the tokenizer decides your bill. How tokenization works, why languages and formats cost more, and how to count your own.
Streaming: Why First-Token Time Beats Total Time
Users forgive long answers that start fast, and never forgive silence. First-token time, streaming mechanics, and the rendering loop that changes the feeling.
Semantic Caching: When the Answer Already Exists
Semantic caching returns stored answers for new questions that mean the same thing. When it pays, the false-hit problem, and the invalidation nobody demos.
The Catalog Moves: Living With Model Versions and Deprecations
Model catalogs move: versions ship, prices change, old ids retire. Pin vs float, what a healthy deprecation policy looks like, and how to never miss one.
Timeouts, Retries, Idempotency: The Resilience Checklist Nobody Writes Down
Every LLM integration fails the same five ways. The unglamorous checklist — timeouts, retries, idempotency, circuit breakers, degradation — in one place.
Latency Is a Feature You Pay For
LLM latency is three numbers, not one: time to first token, per-token speed, total time. What each is ruled by, and which knobs you actually control.
Rate Limits Are an Architecture Input, Not an Error
A 429 is not an exception, it is the contract. Reading rate limits as a capacity plan, the four patterns that absorb them, and what retry storms cost.
The Enterprise Inference Vendor Checklist
A procurement-ready checklist for evaluating LLM inference providers: data policy, pricing transparency, reliability evidence, catalog, and exit terms.
Prompt Caching and the Economics of the Static Prefix
Your system prompt is a subscription paid on every request. Prompt caching turns repeat context into a discounted asset — the mechanics and the math.
Why Hard-Coding One Model Is Now Your Biggest AI Cost Line
Pinning every request to one LLM feels simple. It quietly sets your price ceiling, your quality floor, and your negotiating position.
What "OpenAI-Compatible" Actually Buys You
Every inference provider claims an OpenAI-compatible API. What compatibility actually covers, what it leaves behind, and how to test it in an afternoon.
Embeddings and Vector Search: The Hidden Cost
Embeddings and vector search sit between the question and the answer — and their cost is rarely priced correctly. The hidden line items and the quality tradeoffs.
LLM Observability: What Logs Miss
Logs show what happened; traces show why. The observability stack that catches drift, the dashboards that earn their noise, and what to watch first.
Quantization: Smaller Models, Cheaper Bills
Quantization shrinks models, speeds them up, and cuts the bill — at a quality price. What it does, where it earns its keep, and when to leave the weights alone.
Context Window Management Without Losing the Thread
Every conversation eventually outgrows the context window. Summarization, retrieval, and the other strategies — and what each one costs in fidelity.
Browse other topics