You have a task an off-the-shelf language model doesn't quite nail — answering questions about your internal docs, replying in a fixed format, classifying tickets your way. You have three tools to close the gap without training a model from scratch: prompting, retrieval-augmented generation (RAG), and fine-tuning. The single most common mistake teams make is reaching for the expensive one first, or using one to solve a problem another is built for. Here is how to choose.

The three options, in one line each

• Prompting — change the instructions you send the model. Zero training, instant to iterate, no infrastructure.
• RAG — fetch relevant information at query time and put it in the prompt, so the model answers from your data instead of its memory.
• Fine-tuning — continue training the model on examples so the behaviour itself shifts: tone, format, a narrow skill.

They are not rivals so much as rungs on a ladder. You climb only as high as the problem forces you to.

Start with prompting — always

Prompting is the cheapest, fastest, and most reversible lever, and modern models respond to it far better than most people expect. Before anything else, see how far a clear, well-structured prompt gets you: spell out the task, show two or three worked examples (few-shot), state the output format explicitly, and give the model room to reason where the task is hard.

A surprising share of "we need to fine-tune" projects evaporate here. If a sharper prompt with a couple of examples gets you to good-enough, stop. You have spent an afternoon, not a budget.

Reach for RAG when the problem is knowledge

Prompting hits a wall when the model simply doesn't know the thing — your product catalogue, last week's policy change, the contents of a 200-page contract. No amount of instruction conjures facts the model never saw.

That is RAG's job. You index your documents, and at query time you retrieve the passages relevant to the question and hand them to the model alongside the prompt. The model answers from text in front of it rather than from half-remembered training data. The wins are concrete: answers stay current (update the index, not the model), you can cite sources, and you cut a major class of hallucination because the model is grounded in retrieved evidence.

The tell that you need RAG: your failures are wrong or outdated facts, and the right answer exists in a document somewhere.

Reach for fine-tuning when the problem is behaviour

Fine-tuning earns its cost when the gap is not knowledge but form — you need the model to reliably produce a specific structure, adopt a consistent house voice, follow a niche convention, or perform a narrow classification with high precision that prompting can't make stable. By training on hundreds or thousands of input-output examples, you bake the pattern in, which can also let a smaller, cheaper model match a larger one on that one task.

The tell: your failures are about how the model responds, not what it knows — inconsistent format, drifting tone, a skill it can almost but not quite do reliably from instructions alone.

The mistake to avoid

The classic error is fine-tuning to add knowledge. Teams fine-tune a model on their documents hoping it will "learn the facts" — and get a model that has picked up the style of the documents while still hallucinating the specifics, now with more confidence. Fine-tuning teaches behaviour, not a reliable, updatable fact store. For knowledge, use RAG. Keeping the two jobs separate is most of the skill.

They compose

These are not mutually exclusive, and strong systems often use all three: a fine-tuned model that reliably emits your format, fed retrieved context via RAG, steered by a clear prompt. The ladder still applies — add each layer only when the one below it has run out of road.

A decision order you can actually use

1. Prompt first. Clear instructions, a few examples, explicit output format. Ship if it works.
2. Wrong or stale facts? Add RAG. Ground the model in your documents before you consider training anything.
3. Wrong form, not wrong facts? Fine-tune — on a focused set of examples, ideally to make a smaller model do one job well.
4. Re-evaluate before you escalate. Each rung adds cost, latency, and maintenance. Most production tasks never need to leave the first two.

The instinct to fine-tune signals seriousness, so it feels like the "real" answer. Usually it isn't. The teams who ship fastest are the ones who exhaust prompting, ground with retrieval, and reserve training for the narrow cases that genuinely demand it.