Fine-Tune, Prompt, or RAG? A Decision That Deserves Better Than a Default

Every team that outgrows a stock model faces the same three-way fork, and most take whichever branch they heard about first. The levers are not interchangeable, and the confusion between them is expensive, so here is the cleanest framing we know: prompting tells the model what to do right now, retrieval tells the model what is true right now, and fine-tuning changes what the model is.

Prompt engineering is free to try and instant to iterate. Retrieval-augmented generation bolts a search step onto the prompt, so the model answers from your documents instead of its training data. Fine-tuning trains the model itself on your examples, producing weights you own. The first two live entirely at request time; the third moves the work to training time and pays you back on every subsequent request.

The decision is not "which is best" but "which problem do I actually have" — and most failed projects we see picked a lever that could not, in principle, solve their problem.

Prompting is the cheapest experiment in all of machine learning: no infrastructure, no pipeline, results in minutes. A well-structured prompt with clear instructions, a defined output format, and a few worked examples closes a surprising share of the gap between a stock model and a custom one. Teams routinely reach for training when a tighter prompt would have done it — and then maintain a training pipeline for the rest of the year.

The prompt is also where you learn what the real problem is. If a stock model fails your task, watch how it fails. If better instructions and examples fix it, you never had a training problem. If the model knows nothing about your domain's facts, that is a knowledge gap, and no amount of instruction adds facts — that is retrieval's job. If the model knows the facts and understands the instructions but cannot hold the format, the voice, or the judgment consistently, that is a behavior problem, and behavior at scale is fine-tuning's job.

Treat the prompting phase as diagnosis, not just attempt. Its failures are the specification for whatever you build next.

Retrieval earns its place when the answer depends on knowledge that is proprietary, recent, or both: your documentation, your tickets, your policies, your inventory. The model cannot know these things — they were not in its training data — and retraining every time a document changes is a category error. Retrieval keeps knowledge outside the model, where it can be updated by editing a document instead of running a training job.

The costs are real and should be budgeted: a retrieval pipeline to build and maintain, chunks of retrieved text inflating every prompt, and a new failure mode — the right answer exists in your corpus but the retriever did not find it. The prompt-size side has its own economics, which we covered in prompt caching and the static prefix, and the crossover question between "stuff the documents in" and "retrieve just the relevant pieces" gets its own treatment in long context vs RAG.

What retrieval cannot do is behavior. If the model quotes your documents in the wrong voice, answers in the wrong structure, or misses the judgment calls your experts make, no retriever fixes that. Knowledge is retrieved; behavior is trained.

Fine-tuning is the answer when you need the model itself to be different: a house style that survives any prompt, an output format that never drifts, a classification judgment tuned to your edge cases, a small model performing like a bigger one on your narrow task. Training on your examples moves the behavior from the prompt — where it consumes tokens on every request and varies with every phrasing — into the weights, where it is stable and free at request time.

The economics matter at scale. A long system prompt full of style rules and examples is billed on every call, forever; a fine-tuned model carries the same behavior for nothing per request. Past some volume, the training run pays for itself — and the result is an asset you own outright, not a prompt you rent. On Run BiOS the trained weights are yours, which is the difference between customizing a model and depending on one; the mechanics are in our SFT guide and the training docs.

What fine-tuning cannot do is add living knowledge. A fine-tuned model's facts freeze at training time, and asking training to keep up with a changing knowledge base is the classic misuse. Facts that change belong in retrieval; behavior that must hold belongs in the weights.

Each lever has a characteristic billing signature, and matching it to your traffic shape is most of the decision. Prompting bills forever: the instructions, the examples, the style rules ride every request, so its cost scales linearly with volume and never amortizes. A long, beautiful system prompt is a recurring subscription you pay per token — which is why the prompt caching post treats the static prefix as an asset worth engineering.

Retrieval bills per request plus a standing cost: smaller prompts, but an index to build, refresh, and operate. Its curve starts above zero and rises slowly; prompting starts at zero and rises steeply.

Fine-tuning inverts the shape: a real upfront cost — the training run, the data work, the evaluation — and then requests that are both cheaper and shorter, because the behavior lives in the weights instead of the prompt. The crossover against a fat system prompt is arithmetic you can run on your own volumes, the same break-even logic as serverless vs dedicated applied one layer up.

The pattern to recognize in your own numbers: high volume plus a long prompt is the signature of a workload begging to be fine-tuned; low volume plus a fast-moving knowledge base is retrieval's home turf; everything in the prototyping phase belongs to the prompt.

The framing "pick one" survives almost no contact with a real product. The common production shape is a fine-tuned model behind a retrieval step: the retriever supplies the facts of the moment, the trained weights supply the voice, format, and judgment, and a thin prompt supplies the per-request instructions. Each lever does the job it is good at, and none is asked to impersonate another.

There is also a useful sequence here. Teams that fine-tune first often discover they built the wrong thing; teams that prompt first arrive at fine-tuning with a precise specification of the behavior gap and a folder of real examples — which then becomes the training set. Sizing that set is its own question, taken up in how much data do you need to fine-tune. The diagnosis path and the data-collection path are the same path.

If you take one sentence from this post: spend a week prompting before you spend a dollar training, and let the failures tell you which lever to buy.

More often than the tooling industry admits. If a general model with a decent prompt meets your quality bar, every additional lever is ongoing maintenance for a difference your users cannot feel. The stock model is also the fastest thing to change: when a better model releases, a prompt-only system switches with a config line, while a trained model needs a retrain.

That reversibility is worth protecting, and it is why we keep the serving side OpenAI-compatible — the migration post covers what a two-line switch looks like. Fine-tuning is the commitment you make after the workload has told you, in detail, that stock is not enough.

The fork is not a judgment on ambition. It is sequencing: prompt to learn, retrieve to know, train to be. In that order.