Use RAG when the AI needs to know your facts. Use fine-tuning when it needs to behave differently. Retrieval-augmented generation hands the model the right passages from your documents at the moment it answers. Fine-tuning retrains the model on examples so its style, format or judgment changes for good. For most business assistants, RAG comes first, and fine-tuning is a later optimization, if you need it at all.
What does each approach actually change?
The confusion comes from both being described as "training the AI on your data." Under the hood, they touch completely different parts of the system, and that’s why they fail in different ways.
RAG: the model reads before it answers
A RAG system indexes your content, pulls the few passages that matter for each question and puts them in the prompt with an instruction to answer only from them. The model itself doesn’t change. Update a policy document and the next answer reflects it, with a citation pointing to the exact clause. Our explainer on what RAG is covers the four steps in detail.
Fine-tuning: the model learns from examples
Fine-tuning continues a model’s training on hundreds or thousands of pairs of inputs and ideal outputs. The model picks up patterns: how you format a claim summary, which category an email belongs in, how your brand words a product description. Newer methods like LoRA (low-rank adaptation) make this cheaper than it used to be, but it’s still a training project, with datasets, training runs and versioning.
RAG vs fine-tuning: how do they compare?
This is the comparison we go through with clients on our LLM and RAG development projects before anyone writes a line of code.
RAG vs fine-tuning, criterion by criterion
| Criterion | RAG | Fine-tuning |
|---|---|---|
| What it changes | The context the model sees | The model’s weights and behavior |
| Content that changes often | Strong: update the index | Weak: retrain to update |
| Source citations | Built in | Not available |
| Per-user permissions | Filtered at search time | Not possible inside the model |
| Consistent format or tone | Moderate, through prompts and examples | Strong |
| Narrow classification at volume | Possible, but prompt-heavy | Strong, and often cheaper per call |
| Upfront work | Content prep, indexing, evaluation | Labeled examples, training runs, evaluation |
| Ongoing work | Keeping content current | Retraining as needs or base models change |
| Switching model vendors | Easy: point it at a new model | Hard: the tuning is tied to one model |
RAG
- What it changes
- The context the model sees
- Content that changes often
- Strong: update the index
- Source citations
- Built in
- Per-user permissions
- Filtered at search time
- Consistent format or tone
- Moderate, through prompts and examples
- Narrow classification at volume
- Possible, but prompt-heavy
- Upfront work
- Content prep, indexing, evaluation
- Ongoing work
- Keeping content current
- Switching model vendors
- Easy: point it at a new model
Fine-tuning
- What it changes
- The model’s weights and behavior
- Content that changes often
- Weak: retrain to update
- Source citations
- Not available
- Per-user permissions
- Not possible inside the model
- Consistent format or tone
- Strong
- Narrow classification at volume
- Strong, and often cheaper per call
- Upfront work
- Labeled examples, training runs, evaluation
- Ongoing work
- Retraining as needs or base models change
- Switching model vendors
- Hard: the tuning is tied to one model
Which should you choose? Answer these questions
Run your use case through these questions. If most of your answers point to one column, that’s your starting point.
| Question | If yes, lean toward |
|---|---|
| Does the AI need to answer from documents, records or policies? | RAG |
| Does that content change more than a few times a year? | RAG |
| Do users need to be able to check where an answer came from? | RAG |
| Should different users get answers from different documents? | RAG |
| Do you need the exact same output structure every time, at high volume? | Fine-tuning |
| Is the task a narrow judgment call (classify, route, extract) with thousands of labeled examples? | Fine-tuning |
| Have you tested prompts and RAG, and they still miss a measurable target? | Fine-tuning, on top of RAG |
What does RAG look like in practice?
Here’s an illustrative RAG answer for a fictional property manager. The question comes in, two passages are pulled from the pet policy and a lease addendum, and every sentence of the reply cites its source. No retraining involved: changing the deposit amount next month means editing one document.
Illustrative example Fictional business, people and data. Not a real client, conversation or result.
Three scenarios from US businesses
These are illustrative scenarios based on common requests, not client stories.
A property management company answering tenant questions
Tenants ask about pets, parking, maintenance windows and lease terms, and the answers are different for each building. The policies change every renewal cycle. This is a clear-cut RAG case: index the policies and addenda for each property, filter by the tenant’s building, cite the clause. Fine-tuning would lock last year’s rules into the model.
An insurance agency routing incoming email
Hundreds of emails a day need tagging: new quote, claim, billing, policy change, certificate request. The categories don’t change much, and the agency has years of emails its staff already sorted. A small fine-tuned model can classify these fast and cheaply. RAG doesn’t add much here, though it’s worth benchmarking a prompt-only approach first.
An online store writing product descriptions in its brand voice
The store needs thousands of descriptions in one recognizable voice with a fixed structure. Product facts come from the catalog (retrieval), and the voice and structure come from examples. This is the hybrid case: retrieve the specs, and fine-tune only if a well-written prompt with examples can’t keep the voice consistent.
Why "fine-tune it on our documents" usually disappoints
There are practical costs too. Every content change means a new training run and a new round of evaluation. You can’t enforce permissions inside a model, so a model fine-tuned on HR files will happily share them with anyone. And when a better base model comes out, your tuning doesn’t carry over.
When isn’t RAG enough?
RAG has its own limits, and knowing them keeps you from forcing it where it doesn’t fit:
- Strict output formats at scale. If downstream software needs exact JSON or a rigid template thousands of times a day, structured-output features or a tuned model may be more reliable.
- Specialized vocabulary or judgment. A model that has to classify radiology orders or freight exceptions by your internal codes may need examples baked in.
- Latency and cost at very high volume. Retrieval adds context to every call; a small tuned model can be faster and cheaper for a single repetitive task.
- Questions about numbers. "What was our return rate last quarter?" needs a database query tool, not retrieval or tuning.
In what order should you test them?
We run the same sequence on every project, because each step costs less than the next one and often makes it unnecessary.
The order we test them in
-
Step 1
Write an evaluation set first
50 to 200 real questions or tasks with known good answers, so every option gets measured the same way
-
Step 2
Try strong prompts
clear instructions and a few examples on a capable model; this gets you further than most people expect
-
Step 3
Add RAG for knowledge gaps
index the relevant content, add hybrid search and citations, then run the evaluation again
-
Step 4
Fine-tune only the remaining gap
if a specific behavior still misses the target, tune for that behavior alone, on top of retrieval
-
Step 5
Re-evaluate on every change
new content, new model or new prompt, the same test set decides
How to choose for your project
If you’re deciding right now, start by writing down 20 questions your users actually ask and where the right answer lives today. That list usually settles it. If the answers live in documents, build retrieval. If they live in your team’s judgment and you have thousands of examples of it, consider tuning.
For a closer look at the architecture, our guide on how to build a RAG chatbot walks through it. If the model also needs to take action in your systems, read how to build an AI agent. When the assistant has to plug into your existing software, our AI integration team handles that side. Or send us your use case and we’ll tell you which approach we’d test first, and why.