Rate Limits Are an Architecture Input, Not an Error

Most integrations meet the rate limit the same way: everything works in testing, the launch goes well, and then one busy morning the error rates spike and someone gets paged for a response that says, politely, "you are doing this too fast." The team treats it as an outage. It is not an outage. It is the provider's capacity plan arriving in your error logs, exactly as published.

Rate limits are the terms under which shared infrastructure stays shared: so many requests per minute, so many tokens per minute, sometimes so many concurrent generations. They are not hidden — they are in the docs and often in the response headers of every successful call. The failure is architectural: the limits were never read as an input to the design, so the design discovered them in production.

The fix is not a better try-except. It is treating the limit as a budget you build around, the same way you would treat a database connection ceiling.

Start by learning which currency you are actually limited in. Requests-per-minute ceilings bind chatty workloads with small prompts; tokens-per-minute ceilings bind workloads with long prompts or long answers, regardless of request count; concurrency limits bind workloads with long-running generations. A team can be nowhere near its request limit while sitting on its token limit, and the dashboards that only count requests will swear everything is fine.

Then do the arithmetic the other direction: take your peak minute, not your average day. Average traffic never trips a limit; peaks do. If your product has a morning rush, a marketing campaign, or a nightly batch window, that is the number the limit has to survive. If the peak does not fit inside the published ceiling, the answer is a conversation with the provider before launch, not a retry loop after it — raised limits against committed volume are a normal enterprise discussion, and the questions to ask are in the vendor checklist.

Finally, keep the response headers. Most providers return your remaining budget on every call. Logging it turns rate-limit management from incident response into a gauge on a dashboard.

A queue in front of the client. Your own system accepts work at whatever rate it arrives and releases it to the provider at the rate the contract allows. This converts bursts into smooth flow, which is often the entire problem. As a side effect, a queue makes you ask which work is actually urgent — the same question that unlocks batch pricing.

Backoff with jitter. When a 429 does arrive, retry after a delay that grows with each attempt, randomized so that a hundred rejected clients do not all retry in the same second. Without jitter you do not have a retry policy, you have a synchronized hammer.

A concurrency cap. A hard ceiling on in-flight requests, set below the provider's, so your system discovers its own throttle before the provider's. Your own limit is one you can tune in advance; theirs arrives as an error.

Priorities. When capacity is tight, interactive requests should jump the queue and deferrable ones should wait. If a nightly report can delay an interactive chat answer, the queue is missing a lane.

Here is the part that belongs in a cost discussion: naive retries do not just fail harder, they cost more. A request that fails after consuming its prompt still ran the prefill; a client that retries immediately, at full length, in a loop, is converting a capacity problem into an invoice. Under sustained pressure, retry traffic competes with real traffic for the same limited budget, which produces more 429s, which produces more retries. Teams have watched the majority of their spend during an incident go to requests that never produced an answer.

The defenses are the patterns above plus two disciplines. Idempotency: if a request might have succeeded before the connection dropped, the retry must not bill as a second execution of the same work — use idempotency keys where the API supports them. And a circuit breaker: past some failure threshold, stop retrying, shed load, and fail fast. A system that degrades to a clean "try again shortly" is cheaper and more honest than one that melts down thoroughly. The rest of the failure-handling checklist — timeouts, idempotency, circuit breakers — is in the resilience post.

This is also where deferrable work earns its keep twice: batch traffic can simply wait out the incident, because its deadline has room in it.

A team that has internalized all of this has a short, boring list of things on a dashboard. Queue depth: how much work is waiting for the provider right now, and whether it drains. Rejection rate: how many 429s per hour, plotted as a line that should sit at zero and means something specific when it does not. Remaining budget: the response-header counters, graphed, so an approaching ceiling is visible minutes before it is felt. And retry counts as a first-class metric, because retries are the earliest symptom of every failure mode in this post.

With those four in place, the operational posture changes. Rate limits stop being surprises and start being seasonality — you learn the shape of your own peaks, you watch headroom shrink months before it runs out, and the conversation with the provider happens while it is still a planning discussion. The unmonitored version of the same system finds out from a pager.

None of this requires exotic tooling. It requires deciding that the limits are telemetry, and treating the decision as seriously as the integration itself.

Sometimes the honest finding is that the architecture is chatty in a way the product does not require. Agent loops that call the model for every micro-decision, polling patterns that re-ask the same question, prompts that re-send a growing conversation in full when a summary would do — these multiply requests without multiplying value, and rate limits are simply where the multiplication shows up first.

Before asking for a higher ceiling, audit the request graph: how many model calls does one user action cause, and which of them could be cached, merged, or deleted? Cutting request count is the only rate-limit fix that also cuts the bill.

And when the workload is legitimately large and steady, the ceiling question becomes a capacity question — the point where shared infrastructure stops fitting at any limit is the break-even conversation we walked through in serverless vs dedicated. Rate limits are not a wall to resent; they are the shared tier telling you the truth about your size.

When you are the only user. A prototype, an internal tool with a dozen colleagues, a side project — read the limits once, set a retry with backoff and a cap, and stop. The queueing theory and the dashboards pay for themselves when strangers depend on the thing.

It is also overkill when your peak sits an order of magnitude under the published ceiling and your growth curve says it will stay there. Check the headroom once a quarter, keep the response-header counters in a log line, and spend the engineering time on the product. The goal was never zero 429s as an achievement; it is zero 429s your users caused.

What is never overkill is the two-line version: retries with jitter and a cap, and a timeout on every call. Everything above that is insurance, sized to the traffic you actually have — and bought when the traffic arrives, not before.