LLM Inference Engineering
The engineering side of serving models in production: model routing, prompt caching, API migration, and vendor evaluation. Written for the people who own the endpoint when it pages.
The Catalog Moves: Living With Model Versions and Deprecations
Model catalogs move: versions ship, prices change, old ids retire. Pin vs float, what a healthy deprecation policy looks like, and how to never miss one.
Timeouts, Retries, Idempotency: The Resilience Checklist Nobody Writes Down
Every LLM integration fails the same five ways. The unglamorous checklist — timeouts, retries, idempotency, circuit breakers, degradation — in one place.
Latency Is a Feature You Pay For
LLM latency is three numbers, not one: time to first token, per-token speed, total time. What each is ruled by, and which knobs you actually control.
Rate Limits Are an Architecture Input, Not an Error
A 429 is not an exception, it is the contract. Reading rate limits as a capacity plan, the four patterns that absorb them, and what retry storms cost.
The Enterprise Inference Vendor Checklist
A procurement-ready checklist for evaluating LLM inference providers: data policy, pricing transparency, reliability evidence, catalog, and exit terms.
Prompt Caching and the Economics of the Static Prefix
Your system prompt is a subscription paid on every request. Prompt caching turns repeat context into a discounted asset — the mechanics and the math.
Why Hard-Coding One Model Is Now Your Biggest AI Cost Line
Pinning every request to one LLM feels simple. It quietly sets your price ceiling, your quality floor, and your negotiating position.
What "OpenAI-Compatible" Actually Buys You
Every inference provider claims an OpenAI-compatible API. What compatibility actually covers, what it leaves behind, and how to test it in an afternoon.
Browse other topics