Self-Hosting LLMs: The Full Cost

The self-hosting pitch always starts with the same slide: the GPU price divided by the tokens it can serve, compared against the API's per-token rate. The arithmetic shows the GPU winning by a wide margin, and the conclusion writes itself — why rent the tokens when you can own the machine?

The slide is not wrong; it is incomplete. The GPU is one line item in a system that includes power, cooling, networking, storage, redundancy, and the people who keep it running. The per-token comparison that includes only the hardware is the same category of error as the price-list post warns about — a number that describes no actual purchase.

This post is the full invoice: the line items the slide omits, the utilization math that decides everything, and the honest comparison against the API.

The hardware: the GPUs, the servers around them, and the storage for the model weights and the data. The facility: power, cooling, rack space — the costs that scale with every GPU hour whether the GPU is serving or idle. The network: the bandwidth and the connectivity that make the endpoint reachable. The redundancy: the second machine, the backup power, the failover path — because a single GPU is a single point of failure, and production means the failure is planned for.

Then the people: the engineers who provision, patch, monitor, and fix. The people cost is the line item that never appears on the slide and always appears on the invoice — and it does not scale down when the traffic does.

And the opportunity cost: the capital tied up in hardware that could have been spent on the product. The GPU is a depreciating asset; the product is the thing that compounds.

The ones that surprise first-time self-hosters, in order of surprise. Idle time: the GPU bills every hour, and the traffic arrives in bursts — the machine is paid for at peak and empty at trough, and the trough hours are pure loss. Overprovisioning: the capacity must cover the peak, not the average, so a chunk of the hardware is idle by design. Model updates: every new model version means new weights, new storage, new tuning — the hardware that served last quarter's model serves this quarter's model worse, and the upgrade cycle is a recurring capital expense.

Then the operational surprises: the security patching, the compliance questions, the incident at the weekend. The API provider absorbs these; the self-hoster staffs them.

And the quiet one: the exit. The hardware is bought, the model is deployed, and the market moves — the API prices fall, the new model arrives, and the self-hosted stack is now the expensive option, with the sunk cost arguing against the switch.

The entire self-hosting economics collapse into one number: utilization — the fraction of the GPU's capacity that is actually serving tokens. A GPU running at full utilization is the slide's arithmetic, and it wins. A GPU running at a fraction is the slide's arithmetic divided by that fraction, and it loses.

The utilization is decided by the traffic pattern, not the engineering. A steady, predictable, always-on workload can run a GPU hot; a bursty, spiky, seasonal workload cannot — the capacity that covers the peak idles through the trough. The batch post's argument about pay-per-use applies here with hardware: the spiky workload is exactly the one that should rent.

And the utilization math includes the maintenance windows, the failed runs, the redeploys — every hour the machine is not serving is an hour the slide assumed it was. The honest utilization number is always lower than the planned one.

Three conditions, all of them necessary. The utilization is high and steady: the workload is predictable enough to keep the hardware hot. The scale is large enough that the people cost amortizes: the engineering team is spread across enough tokens that its cost per token is small. And the requirements demand it: data that cannot leave the building, latency that cannot tolerate a network, control that the API cannot offer.

The third condition is the one that actually decides most self-hosting decisions — the requirements, not the arithmetic. The teams that self-host for compliance or control reasons are not comparing token prices; they are buying a property the API does not sell.

And the honest test is the same as every cost decision on this blog: model the full cost, including the people and the idle time, and compare against the API at your actual volume. The slide's arithmetic is the starting point, never the conclusion.

When the workload is spiky, small, or unproven. The bursty traffic pays for idle hardware; the small traffic cannot amortize the people; the unproven traffic is a bet made with capital instead of tokens. The pay-per-use model exists precisely for these workloads, and the batch post's rule applies: unproven workloads deserve pay-per-use.

It is also the wrong answer when the model landscape is moving fast. The hardware bought for this quarter's model is the wrong hardware for next quarter's — and the API customer switches models with a config change while the self-hoster runs a procurement.

And it is the wrong answer when the team is not staffed for operations. The self-hosted stack is a production system with all the production responsibilities — the monitoring, the patching, the on-call. Buying the GPU without buying the operations is how the slide's savings become the invoice's surprise.