Quantization: Smaller Models, Cheaper Bills

A model's weights are numbers, and numbers can be stored at different precisions. The full-precision model keeps every weight at the precision it was trained at; the quantized model rounds them down to fewer bits. The result is a model that is dramatically smaller, runs faster, fits on cheaper hardware, and costs less to serve — in exchange for a quality loss that is sometimes negligible and sometimes not.

The promise is real: quantized models are how frontier-scale capability reaches consumer hardware, and how production systems cut their serving bills. The catch is that quantization is a dial, not a switch — the quality loss depends on the model, the precision, the method, and the task.

This post is the dial: what quantization does, what it costs, and how to decide where the dial belongs for your workload.

It reduces the precision of the weights. A model trained at full precision stores each weight in a wide numeric format; quantization maps those values onto a smaller format — fewer bits per weight — so the model occupies a fraction of the memory and the arithmetic runs faster on hardware that prefers small numbers.

The mapping is the craft. The naive approach rounds every weight to the nearest representable value; the sophisticated approaches measure the weights' distribution first and map them so the values that matter keep their precision. The difference between naive and careful quantization is the difference between a model that degrades gracefully and one that falls off a cliff.

And quantization applies to the activations too, not just the weights — the intermediate values computed during inference. Quantizing both is where the speed comes from; quantizing only the weights is where the safety is. The methods differ in how far they push each side.

It depends on three things, and the honest answer is always "measure it". The precision: the fewer bits, the more the model forgets — the drop from full precision to a moderate quantization is often small, and the drop to an aggressive one is often large. The model: larger models tolerate quantization better than smaller ones, because they have redundancy to spare. And the task: factual recall and exact reasoning degrade faster than fluency and style, because precision loss hits the details first.

The measurement is the eval discipline: run the same task set through the full model and the quantized model, and compare. The comparison is the only answer that matters — every rule of thumb about quantization is a rumor until your eval confirms it.

And the failure mode to watch is not the average but the tail. A quantized model can post a fine average while failing the cases that needed the lost precision — the long-tail degradation that the average hides.

The quantization ecosystem has a format for every purpose, and the names matter less than the properties. The formats differ in what they quantize — weights only, or weights and activations — and in how they choose the mapping — per-layer, per-group, or per-value. The tradeoff is always the same: more aggressive quantization buys more speed and risks more quality.

The practical question is compatibility: which formats your serving stack supports, and which the model you want is available in. A quantization method that is brilliant in the paper and absent from your serving stack is a research result, not an option.

And the practical discipline: pick the format your stack supports well, quantize at the moderate setting first, and eval. The aggressive settings are for when the moderate one passes the eval and the bill still hurts.

Three places, in rising order of stakes. Local and edge deployment: the model must fit on the hardware, and quantization is the difference between running and not running — the quality tradeoff is made for you. High-volume serving: the same answers at a fraction of the serving cost, when the eval shows the quality holds — the cost-per-task math improves without touching the model choice. And latency-sensitive paths: the smaller model runs faster, and the first-token time improves — the perceived-latency win without the streaming work.

The common thread: quantization is a cost lever, and like every cost lever on this blog it is pulled against the eval. The eval says whether the quality held; the invoice says whether the savings mattered.

And the compounding effect: a quantized model that passes the eval can also unlock a cheaper serving tier, a smaller GPU, a higher concurrency — the savings multiply beyond the raw speedup.

When the task cannot tolerate the precision loss and the eval says so. Exact reasoning, long-horizon planning, anything where the tail cases are the product — the quantization savings are not worth the silent degradation. The eval is the arbiter, and a failed eval is a complete answer.

It is also the wrong move when the model is small already. Quantizing a small model costs proportionally more quality, because there is less redundancy to spare — the savings shrink and the risk grows. The lever works best on the models that are expensive to serve, not the ones that are cheap.

And it is the wrong move when the serving stack does not support the format well. A quantization that runs slower than the full model because the hardware does not like the format is a downgrade in every dimension. The format must fit the stack, not just the paper.