RAFT: Adapting Language Model to Domain Specific RAG

When a model will answer questions about a known set of documents (a company's manuals, an API's docs, a medical corpus), there are two usual choices. Retrieval-augmented generation leaves the model as it is and pastes retrieved passages into the prompt. Fine-tuning teaches the model the domain but usually trains on questions and answers without documents. Tianjun Zhang, Shishir Patil, Joseph Gonzalez and colleagues at UC Berkeley framed it as exam preparation: RAG is an open-book exam without studying, fine-tuning is studying for a closed-book exam. RAFT (Retrieval-Augmented Fine-Tuning) studies for the open-book exam.

Each training row has a question, a set of documents and an answer. Documents are either golden (the answer can be deduced from them) or distractors (they cannot). For a fraction P of questions, the context contains the golden document and k distractors (four in most experiments). For the remaining 1 − P, it contains only distractors. The target answer is the same in both cases: a chain-of-thought explanation, written by GPT-4, that quotes the golden document between ##begin_quote## and ##end_quote## markers and ends with the answer. Llama 2 7B is fine-tuned on these rows. At test time, it gets a question and retrieved documents, like any RAG system.

The authors give two reasons. Distractors teach the model to ignore irrelevant retrieved text, and rows without the golden document push it to learn domain knowledge rather than only reading. They evaluated on PubMedQA, HotpotQA and three API-call benchmarks from Gorilla (HuggingFace, Torch Hub, TensorFlow Hub), against the base Llama 2 chat model with and without retrieval, domain-specific fine-tuning (DSF) with and without retrieval, and GPT-3.5 with retrieval.

Customers with a fixed corpus and a narrow task regularly ask whether to fine-tune or use RAG. RAFT is the most cited argument for doing both, and its recipe is easy to reproduce.

Train with the noise you will see. If production retrieval returns one useful passage among several useless ones, training examples should look like that. Models trained only on clean context learn to trust whatever is in front of them.

Answer format is half the win. The base Llama 2 scored 0.03 on HotpotQA with retrieval; fine-tuning on the domain's answer style alone took it to 6.38. Much of what any fine-tune buys is the right output format, which a good prompt or a stronger model can also buy.

Check that quotes are real. RAFT's targets quote the golden document. In rows where that document is absent, the target still quotes it, which teaches the model to produce quotations it cannot see. In production, verify every quotation against the retrieved text.

A strong general model may be enough. GPT-3.5 with retrieval beat RAFT 7B on HotpotQA. For general-knowledge questions the case for fine-tuning a small model is weaker than for specialized API documentation, where RAFT's margins were largest.