Distillation: Teaching Small Models From Big Ones

Distillation is the oldest trick in the model-compression playbook, and it works on language models the same way it worked on everything else: a large, capable model — the teacher — generates outputs, and a small model — the student — is trained to reproduce them. The student never sees the teacher's weights; it sees the teacher's behavior, and it learns to imitate it.

The result is a small model that carries a surprising fraction of the large model's capability, at a fraction of the serving cost. The student is cheaper to run, faster to answer, and easier to deploy — and it has absorbed the teacher's patterns, not just its answers.

The catch is the same as every compression: the transfer is lossy. The student learns what the teacher's outputs reveal, and the teacher's outputs do not reveal everything. The art of distillation is knowing what transfers and what does not.

The teacher generates, the student trains. The training data is the teacher's outputs: prompts are fed to the teacher, the teacher's responses are collected, and the student is fine-tuned on those prompt-response pairs. The student learns to produce the teacher's answers — and, more importantly, the teacher's style of answering: the reasoning steps, the formatting, the tone.

The richer variants transfer more than the final answer. Instead of only the teacher's chosen response, the student trains on the teacher's full output distribution — the probabilities the teacher assigned to every candidate token. That distribution carries the teacher's uncertainty and its near-misses, and the student absorbs a deeper signal than a single answer provides.

And the modern variant adds the reasoning trace: the teacher's step-by-step thinking is included in the training data, so the student learns not just what the teacher concluded but how it got there. The reasoning is often the most valuable thing that transfers.

What transfers well: style, formatting, reasoning patterns, and the teacher's behavior on the tasks represented in the training data. The student picks up the teacher's voice and its approach — the things that show up in the outputs.

What transfers poorly: the teacher's depth of knowledge beyond the training distribution. The student learns the answers it was shown; it does not inherit the teacher's ability to reason about things it was never shown. The distilled model is brilliant on the covered tasks and ordinary everywhere else — the coverage of the distillation data is the ceiling of the student.

And what does not transfer at all: the teacher's uncertainty calibration, its refusal boundaries, its safety behavior — unless those are deliberately included in the training data. A student trained only on confident answers becomes confidently wrong; a student trained without refusal examples never learns to refuse.

The quality of the distillation is the quality of the teacher's outputs, and the volume of the distillation is the breadth of the student. The data question is both: enough outputs to cover the tasks the student must handle, and good enough outputs that the student learns the right patterns.

The generation cost is the tuition: every training example is a teacher call, and a broad distillation is a large teacher bill. The economics are the same shape as every training decision on this blog — the tuition is paid once, the student's savings compound forever, and the crossover is arithmetic.

And the data must include the failures. A student trained only on the teacher's successes never learns what the teacher refuses, where it hesitates, when it says "I cannot". The failure examples are where the safety behavior lives, and omitting them is how distilled models get their reputation for overconfidence.

When the task is narrow enough that the teacher's breadth is wasted. A student distilled for one product's tasks — the support answers, the classification, the extraction — can match the teacher on those tasks at a fraction of the cost, because it does not need the teacher's everything-else.

It also wins on the operational axes: the latency, the deployment footprint, the serving cost. The distilled model runs where the teacher cannot — on the edge, in the tight loop, at the high volume. The capability tradeoff buys the operational fit.

And the honest comparison is the eval: the student against the teacher on the product's actual tasks. The eval decides whether the compression was worth it — the same discipline as every model choice on this blog, with the teacher as the control group.

When the task needs the teacher's full depth. Open-ended reasoning, novel problems, anything where the training distribution cannot cover the real traffic — the student's ceiling is the data, and the data cannot cover what has not happened yet.

It is also the wrong choice when the teacher itself is the product. If the differentiation is the frontier capability, the distilled student is a cheaper imitation of the thing you sell — the savings come out of the product.

And it is the wrong choice when the tuition exceeds the savings. A distillation that requires an enormous teacher bill to cover a small task set is a purchase, not an investment — the crossover math decides, and it is run before the generation starts, not after.