What Serverless LLM Inference Actually Costs at Enterprise Volume

Every inference bill has a crossover hiding in it. Below a certain sustained load, paying per token is the obviously correct choice. Above it, a dedicated endpoint quietly becomes cheaper, and every day you do not notice is a day of overspend. Most writing on this topic explains what serverless inference is and stops before the part that matters. This piece is about the line itself: how to find it for your workload, why the naive calculation puts it in the wrong place, and what actually changes when you cross it.

We sell both shapes of capacity, so we have no interest in pretending one is universally better. What we see across teams is simpler: almost nobody has done the arithmetic, and the ones who have usually did it with the wrong inputs.

Serverless inference bills per token: a published rate per million input tokens and per million output tokens, multiplied by what you actually send and receive. Nothing more. When traffic stops, the bill stops. There is no capacity to reserve, no minimum commitment, and no charge for the machine sitting idle at three in the morning.

The rate card is the whole story, which is what makes it so easy to reason about. Our model library lists every model with its per-token rates and context window, and the same prices apply whether you send ten requests a day or ten thousand. The economics are linear by design: cost tracks usage exactly, with zero waste at the bottom and zero discount at the top.

That linearity is the feature at low and spiky volume. It is also, eventually, the problem.

A dedicated endpoint inverts the shape: you pay for a GPU by the hour, and in return it is entirely yours. The meter runs whether the GPU is answering requests or waiting for them, but the throughput ceiling is yours alone, latency is steadier because no one else is sharing the hardware, and the cost per token falls toward zero as utilization climbs.

Our deployments documentation covers both shapes as they exist on Run BiOS: serverless by default, dedicated clusters for sustained load. Most providers offer some version of the same split, under different names.

The pricing consequence is what matters here. Serverless is a line through the origin: zero usage, zero cost. Dedicated is a flat horizontal line: the same cost at any usage. Where those two lines cross is the only number in this entire discussion, and it is different for every workload.

The formula fits in one sentence: dedicated wins when the tokens you could push through the GPU per hour, multiplied by the serverless per-token price, exceed the hourly rate of the GPU. In symbols, with tokens-per-hour as T, the serverless price per token as p, and the hourly cost as h, the crossover is where T times p equals h. Below it, serverless. Above it, dedicated.

Three variables, and two of them are on your rate card. The one that is not — the throughput a GPU actually sustains on your model, your context lengths, your output lengths — is the one everyone guesses, and guessing it is where the crossover ends up in the wrong place. Throughput is not a spec-sheet number. It depends on how long your prompts are, how long your completions run, and how aggressively requests can be batched.

The honest way to find it is to measure your own traffic pattern for a week, then run the arithmetic with real rates rather than marketing ones. Our pricing calculator exists for exactly this: pick a model, dial in your volume, and it prices the same work across providers, so the serverless side of the line is at least grounded in published numbers rather than a pitch deck.

The first mistake is averaging. Teams take their daily token count, divide by the hours in a day, and compare that average rate against a GPU. But traffic is never average. A support product peaks during business hours and idles overnight; a batch pipeline is dead quiet all day and then saturates everything at midnight. A dedicated GPU sized for the peak wastes money all night, and one sized for the average falls over at noon. The average is precisely the wrong input.

The second mistake is forgetting the shape of the bill you already have. Retries, failed generations you still paid for, and oversized prompts all inflate the token count before they ever reach a pricing formula. If your usage is padded with waste, the crossover you calculate is the crossover for a bloated workload. Clean the workload first; we wrote about the usual suspects in our invoice teardown.

The third mistake is treating latency as free. Shared capacity has scheduling variance, and cold starts exist on every serverless platform whether or not the marketing page mentions them. If your product has a hard latency floor, part of your traffic is effectively pinned to dedicated capacity regardless of what the token math says, and the honest calculation prices that traffic separately.

At small volume, the decision is a rounding error and you should not spend a week on it. At enterprise volume, the decision compounds daily, and three things change.

First, the blend matters more than the total. Enterprise workloads are rarely one workload: interactive chat beside document pipelines beside classification jobs. Each has its own crossover, and the right answer is usually a split — bursty interactive traffic on serverless, sustained batch windows on dedicated — rather than a single shape for everything.

Second, commitments enter the picture. Sustained volume is what gives you leverage on reserved or committed capacity, and a team that has never measured its baseline cannot negotiate one. Measurement is not just for the comparison; it is the prerequisite for any pricing conversation.

Third, the decision stops being static. Model prices move, your traffic grows, and the crossover moves with them. A calculation done once and filed away is a calculation that will be wrong by the next quarter.

Sometimes, and the honest answer matters more than the reassuring one. Shared capacity means shared scheduling: when many tenants burst at once, individual requests wait their turn, and the tail of your latency distribution stretches. Platforms that batch requests for efficiency — most of them, because batching is what makes per-token economics work — add a small, deliberate delay to let batches form. You are trading a little time for a lot of utilization, whether you know it or not.

Cold starts are the other tax. A model that has not served traffic recently has to be loaded before it can answer, and that load time lands on whoever asks first. Providers differ widely in how aggressively they keep models warm, but no shared platform escapes the physics.

Whether any of this matters depends entirely on your product. A document pipeline that runs overnight will never notice scheduling variance. A chat interface with a hard responsiveness floor will notice immediately. This is not a reason to avoid serverless — it is a reason to price the split honestly, which is exactly what the crossover calculation is for.

Pull a week of real usage from your logs: requests per hour across the day, average prompt and completion lengths, and the split between interactive and batch work. Cost the serverless side from published per-token rates — the pricing calculator does this directly from your volume, and the model library has the underlying rates if you want the raw figures. Then get a real hourly quote for a dedicated endpoint that can actually serve your peak, not your average.

If the dedicated quote only beats serverless when the GPU runs hot around the clock, you do not have a dedicated workload yet — you have a spiky one, and spiky workloads are what per-token billing exists for. If your baseline load alone clears the crossover before you even count the peaks, the dedicated endpoint pays for itself and the peaks are gravy.

The answer is rarely all one thing. Price the split, not just the extremes.

If you already own GPU capacity under a long-term commitment, your marginal cost of running one more token on it is close to zero, and no per-token platform can compete with hardware you have already paid for. Use us for the burst above your baseline instead of pretending the baseline does not exist.

If your traffic is a single, sustained, predictable stream that never sleeps, a dedicated endpoint — ours or anyone else's — will beat per-token pricing on pure arithmetic. We will happily sell you that instead; the deployments docs cover the dedicated path.

And if you have not measured your traffic yet, the wrong choice is deciding at all. Measure first. The crossover is a measurement, not a preference.