Almost every client conversation that starts with "we want to fine-tune a model on our data" should have started with "our model does not know about our data." Those are different problems with different solutions, and confusing them is the most expensive mistake in applied AI right now.
The one-line distinction
RAG changes what the model knows. Fine-tuning changes how the model behaves.
If your support bot cannot answer questions about your refund policy because it has never read your refund policy, that is a knowledge gap. Fine-tuning it on a thousand support transcripts will teach it to sound like your support team while still not knowing your refund policy. It will now be wrong in your house style.
If your support bot knows everything but writes six-paragraph essays when your team writes three crisp lines, that is a behaviour gap — and it is probably fixable with a prompt and three examples before you go anywhere near training.
Start with the option nobody blogs about
Before either technique: can you just put the material in the prompt?
Modern context windows are large. If your entire product documentation is 40 pages, you do not need a vector database, an embedding pipeline, a chunking strategy or a re-ranker. You need to paste 40 pages into the context, cache it, and ask the question. It will be more accurate than a naive RAG pipeline, because nothing is being dropped by a retrieval step that guessed wrong.
The math changed. Prompt caching means the repeated cost of a long, stable context is a fraction of the list price. For a lot of small and mid-size businesses, "retrieval" is a solved problem because there is nothing to retrieve from — the whole corpus fits.
Graduate to real retrieval when the corpus stops fitting, when it changes constantly, or when different questions genuinely need different slices of a large body of material.
When it is RAG
Reach for retrieval when any of these are true:
- The knowledge changes. Prices, inventory, policies, tickets, this week's schedule. Re-training on Tuesday for Wednesday's data is not a strategy.
- You need citations. Retrieval hands you the source document for free. A fine-tuned model cannot tell you where it learned something.
- The corpus is large. Thousands of documents, most irrelevant to any given question.
- Access control matters. Retrieval lets you filter documents by who is asking. Weights cannot forget things for one user.
And the part everyone gets wrong
When people say "we tried RAG and it hallucinated anyway", they almost never have a model problem. They have a retrieval problem. The model was handed three irrelevant chunks and did its best.
The quality of a RAG system lives almost entirely in the layer before the model:
- Chunking. Splitting on a fixed character count cuts tables in half and orphans headings. Split on document structure — sections, headings, list boundaries — and keep the heading with the chunk so a fragment still knows what it is about.
- Hybrid search. Pure vector similarity is bad at exact identifiers, part numbers and rare terms. Run keyword search alongside it and merge. This one change fixes more RAG systems than any model upgrade.
- Re-ranking. Retrieve twenty candidates, then use a cheap cross-encoder or a small model to pick the best four. Precision at the top of the list is what the generator actually consumes.
- Show your work. Log the retrieved chunks with every answer. When a user reports a wrong answer, you want to see immediately whether the right document was even in the room.
When it is fine-tuning
Fine-tuning is real, and it is genuinely the right answer for a narrower set of problems than the discourse suggests:
- Format and style you cannot describe. Some outputs — a specific legal clause structure, a house tone, a bespoke markup format — take a thousand words of prompt to approximate and three hundred examples to nail.
- Making a small model punch above its weight. This is the strongest modern case. Prove the task with a frontier model, generate training data from its outputs, then fine-tune a small cheap model to do that one job at a fraction of the cost and latency.
- Classification at volume. If you are running one narrow classification a million times a month, a tuned small model beats a prompted large one on every axis.
- Latency floors. When a big model's response time is a product problem, not a cost problem.
The prerequisites are unforgiving: a few hundred consistent, high-quality examples; a task that will not change next quarter; and an eval set to prove the tuned model actually beat the prompted one. If you do not have all three, you are not ready, and the prompt is cheaper.
The decision, compressed
| The symptom | The fix |
|---|---|
| "It does not know our products / policies / data" | Retrieval (or a big prompt) |
| "It makes things up when it does not know" | Retrieval + an explicit "I do not know" escape hatch |
| "It cannot cite sources" | Retrieval |
| "The information changes weekly" | Retrieval |
| "It does not follow our format" | Prompt + examples first, then fine-tuning |
| "It is too slow / too expensive at our volume" | Fine-tune a small model, or route by difficulty |
| "It is 90% right and we need 99%" | Neither — you need evals, then targeted fixes |
The honest ordering
Almost every project we have shipped follows the same escalation, and almost none reach the end of it:
- Good prompt, with the source material pasted in. Most projects stop here.
- Prompt + real retrieval, when the corpus outgrows the context.
- Retrieval + a re-ranker, when precision matters.
- Fine-tune a small model on the now-proven task, when cost or latency demands it.
Skipping to step four because it sounds like the serious engineering choice is how six-week projects become six-month ones. The unglamorous steps are where the accuracy is.
Not sure which one your problem needs?
We scope AI systems honestly — including telling you when the answer is "a better prompt and a weekend", not a platform.
Ask us →