How Much Data Do You Need to Fine-Tune?

The question arrives in every evaluation call: how many examples do we need? The honest answer is unsatisfying — it depends on the task, the method, and the model — but it is not a shrug. There is a shape to the answer, and the shape is more useful than any number.

What the field has learned, consistently: the count matters far less than what is in the set. A small dataset of excellent, representative examples routinely outperforms a much larger pile of inconsistent ones, because fine-tuning is teaching behavior, and behavior is learned from the pattern in the data, not the volume of it. Noise at scale is still noise — it is just expensive noise.

So this post will not give you a threshold. It will give you the variables that decide yours, the point where adding data stops paying, and a way to find your own answer with experiments small enough to run this week.

Task narrowness is the biggest variable. Teaching a model one output format, one classification scheme, one voice — a single, well-defined behavior — is a small job. Teaching it a domain's worth of judgment is a large one. The more the desired behavior overlaps what the base model already does, the less data you are buying.

The method changes the scale too. Adapter methods like LoRA learn a compact set of new parameters and get remarkable mileage from modest datasets; full fine-tuning rewrites more of the model and wants correspondingly more evidence; continued pre-training, which teaches knowledge rather than behavior, is the hungriest of all. The method ladder and its trade-offs are mapped in the adapter comparison and the SFT guide.

And consistency multiplies everything. If your examples agree with each other — same format, same judgment calls, same conventions — the model finds the pattern quickly. If they were scraped from different eras, authors, and standards, you are not training one behavior, you are training an argument, and no dataset size fixes an argument.

Fine-tuning is supervised imitation: the model learns to produce more of whatever the dataset demonstrates. That makes the dataset a specification, and every example a line in it. Contradictory lines produce contradictory behavior; sloppy lines produce confident sloppiness. This is why the dataset preparation guide spends so much of its length on cleaning and curation — the preparation work is not preprocessing, it is the training.

The practical consequence: before you scale a dataset up, read it. A review pass over your examples will find the mislabeled rows, the format drift, the examples that teach exactly the behavior you are trying to remove. Removing the worst slice of a dataset improves the model more reliably than adding a fresh slice, and it is cheaper in every currency.

If you remember one heuristic: a dataset you would be proud to publish as your company's style guide is ready; a dataset you have never read is not, whatever its size.

The return curve on examples is steep at the start and flattens — the first examples teach the format and the voice, the next fill in common cases, and from there each additional batch adds rarer and rarer coverage. Past the knee of that curve, more of the same data mostly teaches the model what it already learned, while evaluation gets slower and training gets longer.

You can find your own knee empirically: train on a small slice, evaluate on a held-out set, double the data, evaluate again. When the gain from doubling shrinks below what the extra training and curation cost, you have found your answer — for this task, this method, this model. It will be different for the next task, which is why no blog post can hand it to you.

The experiments are deliberately cheap: short runs on small slices, billed per second on whatever GPU tier the method needs, so the search for the knee costs a fraction of the training run it informs.

Three sources, in rough order of value. Your own production logs are the gold standard: real inputs, and — where your product has corrections, edits, or thumbs — real evidence of what the right answer was. Teams sitting on months of logged traffic are usually much closer to a training set than they think; the work is curation, not creation.

Subject-matter experts writing examples is the second source: slower and more expensive per example, and worth it precisely for the judgment cases that logs cannot show you. A few dozen expert examples of the hard cases often move a model more than a mountain of routine ones.

Synthetic data — having a strong model generate examples — is the third, and it comes with an honest warning: you are training your model to imitate another model, biases and blind spots included. Synthetic data is excellent for volume around a pattern you have already verified, and poor as the source of the pattern itself. Generate with a model, but filter with a rubric, and keep a human reading samples from every batch.

The dataset preparation guide covers formats, validation, and import; the sizing question and the sourcing question are the same question seen from two sides.

Sometimes the right answer to "how much data" is "more than you have, and differently than you think" — because the gap you are trying to close is a knowledge gap, and fine-tuning is the wrong tool for it. If the model fails because it does not know your facts, retrieval is the fix, and the full decision is walked through in fine-tune, prompt, or RAG.

"Not yet" is also the answer when your examples exist only as intentions. Teams regularly schedule a training run before the dataset exists, and the dataset work expands to fill months. Build the first small set by hand, train on it, and let a real model — however imperfect — tell you what the next thousand examples should look like.

And if the task is still moving, freeze the target before you count. A dataset sized for last month's definition of the task is the wrong size by construction.