Timeouts, Retries, Idempotency: The Resilience Checklist Nobody Writes Down

Model APIs are remote procedure calls to a very large, very shared computer, and they inherit every failure mode of distributed systems plus a few of their own. The request that hangs forever. The response that is cut off mid-JSON. The timeout that fires after the model already produced the answer you are about to pay for twice. The burst of failures that turns out to be a rate limit. And the slow degradation — answers getting later and later — that never trips an error at all.

None of these are exotic, and none are the provider being uniquely unreliable. They are what production traffic looks like, everywhere, on every vendor. What differs between teams is whether each failure was a design input or a surprise.

This post is the checklist we wish someone had handed us before the first launch. It is deliberately boring: nothing here is clever, and all of it is load-bearing. Rate limits got their own treatment in the rate-limit post; everything else fits on one page.

An HTTP client's default timeout is written for a fast web service, not for a model that may legitimately think for a minute. Two consequences follow, and both bite. Set the timeout too short and you cancel successful requests mid-flight — and keep paying for a completion you never read. Leave it at the default-infinity some clients ship and a wedged connection holds a worker forever, which is how one slow upstream becomes your outage.

The right value comes from your own latency data, not from the provider's marketing: take the tail of your real total-time distribution — the slowest requests your product tolerates, as we argued in the latency post — and set the ceiling a notch above it. Interactive paths get an aggressive ceiling because a user is waiting; batch paths get a generous one because nobody is.

Then log cancellations separately from failures. A timeout is not an error from the provider; it is your own policy firing, and mixing the two makes both invisible.

The dangerous retry is the one after an ambiguous failure: the connection dropped after the request arrived, and you cannot know whether the model answered. Retry naively and a completed generation runs twice — double the tokens, and, if the call had side effects downstream, double the effect. The fix is idempotency: send a stable key with the request so the provider can recognize a duplicate and return the first result instead of running it again. Where the API supports idempotency keys, use them on every mutating call; where it does not, make the retry safe on your side by checking whether the work already landed.

The mechanics of the retry loop itself are settled engineering: exponential backoff, full jitter, a hard cap on attempts. The details and the anti-patterns are in the rate-limit post's section on retry storms — the short version is that a retry policy without jitter and a cap is a way of turning a blip into a self-inflicted denial of service.

And classify before you retry. A request rejected for content will fail forever; a request rejected for capacity will succeed later. Retrying the first is pure spend.

When failures cross a threshold — not one bad request, but a run of them — the correct behavior is to stop calling and start failing fast. That is the entire idea of a circuit breaker: an open circuit returns an immediate, honest error to your users instead of queueing them behind a dead dependency. Your error page is cheaper than your timeout, in latency and in tokens.

The better version degrades instead of failing. Queue the deferrable work for later — the batch lane from the cheapest request is the one that can wait absorbs provider incidents almost for free. Serve a cached or simplified answer where one exists. Fall back to a second model if your abstraction allows it, which is the deepest argument for the two-line-switch architecture in the migration post: an alternative you have already integrated is the only fallback that exists when it matters.

Whatever the fallback, decide it before the incident. "What does the product do when the model is down" is a product question, and the worst time to hold that meeting is during the outage.

The nastiest failure mode returns a perfectly valid response that is silently incomplete. The model hit the output cap, or the stream dropped near the end, and what you receive is most of an answer — which, if the consumer is a parser, is a malformed document that looks like a bug in your code.

The API tells you, if you look: every completion carries a finish reason, and "length" means the answer was cut by the cap, not completed. Check it. A response that ended on length is not a successful response, and treating it as one ships truncated answers to users with a green status code attached.

For structured output, add a parse gate: the response is not done until it parses and validates against the schema. That single check converts a whole class of silent corruption into an ordinary, retryable failure — which the rest of this checklist already knows how to handle.

For a prototype with one user, almost all of it. Wire a timeout so nothing hangs forever, log failures somewhere you will actually look, and build the product. The checklist pays for itself when other people depend on the thing.

The trigger points are boring and predictable: a paying customer, a scheduled job nobody watches, a second engineer who will get paged. Any one of those means the next failure will be somebody's bad morning, and the morning is cheaper to prevent than to attend.

What is never overkill is the one-line version: every call has a timeout, every retry has a cap, every failure is logged with enough context to reconstruct it. That much fits in the first commit, and it is the difference between debugging an incident and narrating one.