Fine-tuning vs RAG vs Prompting

TL;DR: Prompting changes what you ask, RAG changes what the model can see, and fine-tuning changes how the model behaves. Try them in that order because that is also cheapest to most expensive. The cleanest test: if the gap is missing or changing facts, reach for RAG; if it is consistent behavior, tone, or output format, reach for fine-tuning; most often you use prompting plus one of the other two together.

The three levers

• Prompting. You shape behavior with instructions, examples, and format constraints in the request itself. Zero build cost, instant iteration, no infrastructure. It is always the first thing to try and often the only thing you need.
• RAG. You retrieve relevant documents at query time and put them in the prompt, so the model answers from real sources. This is how you give a model knowledge it never had or that changes, and how you make answers citable and auditable.
• Fine-tuning. You continue training the model on your own examples so its default behavior shifts. This is for teaching a style, a format, a domain dialect, or a task pattern that examples in the prompt cannot pin down reliably. It bakes behavior into the weights; it does not give the model live facts.

The axes that decide it

Run the problem through these and the choice usually falls out:

• Facts vs behavior. Need the model to know something specific or current? That is RAG (or prompting, if you can paste it). Need it to consistently act or format a certain way? That points to fine-tuning. This axis alone resolves most cases.
• Data freshness. Knowledge that changes (prices, inventory, policies, this quarter's docs) cannot live in frozen weights. RAG, because you update the index, not the model.
• Auditability and citations. If a regulator or a user needs to see the source for an answer, RAG gives you that; fine-tuning hides knowledge inside weights you cannot point at.
• Data volume. A handful of examples fits in a prompt. Thousands of consistent examples that define a behavior justify fine-tuning.
• Cost and latency. Prompting is free to change but spends tokens every call. RAG adds a retrieval step and more prompt tokens. Fine-tuning has up-front training cost and re-training when behavior needs to change, but can cut per-call tokens by moving instructions into the weights.

Side by side:

A decision sketch

The spine below is the same order as a ladder you climb one rung at a time, with the measurement step the decision sketch leaves out.

They compose

The framing as a three-way fight is the rookie error. In practice they stack. A mature deployment often fine-tunes a model to reliably output the right JSON structure and house tone, uses RAG to feed it current, citable facts, and wraps both in a carefully engineered prompt that sets the rules and the refusal behavior. Fine-tuning handles how it answers, RAG handles what it answers about, prompting orchestrates the request. Asking "RAG or fine-tuning?" as if you must pick one usually means the question has not been split into facts versus behavior yet.

Why interviewers probe this

This is the single most common applied-AI design question because it is the decision you will make on day one of a deployment. They are screening for whether you reach for the cheap lever first and whether you can articulate the facts-versus-behavior split. The failing answer is "we'll fine-tune the model on the customer's documents" to add knowledge, which is slow, expensive, hard to update, and impossible to cite, exactly the job RAG does better. The follow-up they hold back: "the customer's data changes daily, does fine-tuning still make sense?" Strong answer: no, frozen weights cannot track daily changes, so facts go through RAG and you reserve fine-tuning, if anything, for stable behavior and format.

Common misconceptions

• "Fine-tuning teaches the model new facts." It mostly shapes behavior and format and bakes in static patterns; it is a poor and stale way to inject knowledge that changes. Use RAG for facts.
• "RAG and fine-tuning are alternatives." They solve different problems (what the model sees vs how it behaves) and are routinely used together.
• "Fine-tuning is the serious, grown-up option." It is the most expensive to build and maintain. Prompting and RAG solve most problems; fine-tuning is justified by behavior, format, scale, or latency needs, not prestige.
• "More fine-tuning data is always better." Quality and consistency of examples matter more than volume, and a small clean set can beat a large noisy one.

Key takeaways

• Prompting changes the request, RAG changes what the model can see, fine-tuning changes how it behaves; try them in that order, cheapest first.
• Split the problem on facts versus behavior: missing or changing facts go to RAG, consistent behavior or format goes to fine-tuning.
• Use RAG, not fine-tuning, for knowledge that changes or must be cited; frozen weights cannot stay fresh or be audited.
• They compose. Real systems combine a tuned model, retrieved context, and a structured prompt rather than choosing one.