Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Large language models learn a remarkable amount of world knowledge during training, but they keep it in their weights. You cannot look up where an answer came from, you cannot correct one wrong fact without retraining, and the model will confidently invent things it does not know.

The team at Facebook AI Research proposed giving the model a second memory it can search. For each input, a retriever looks up the most relevant passages in an index, here 21 million 100-word chunks of Wikipedia, and a generator writes the answer conditioned on both the question and those passages. They call the weights parametric memory and the index non-parametric memory, and they named the combination retrieval-augmented generation: RAG.

It worked. RAG set new best published results in 2020 on three open-domain question answering benchmarks, beating both models that answer from memory alone and pipelines that extract answer spans from retrieved text. Human raters judged its generated text more factual and more specific than the same generator without retrieval.

The experiment that matters most to practitioners is small. The authors built one index from a 2016 Wikipedia dump and one from 2018, then asked who held 82 positions whose holders had changed. With the matching index, RAG answered about 70% correctly. With the wrong year's index, it fell to 4% to 12%. Change the documents and you change what the model knows.

Most enterprise AI projects are retrieval projects. The customer's knowledge lives in documents, tickets and databases, not in any model's weights, and this paper is where the pattern started.

The index is the knowledge. The hot-swap experiment is the paper's real gift to practitioners. Accuracy did not come from the model; it came from whether the index matched the question. A stale index fails silently: the system still answers fluently, from last year's policy. Keeping the index fresh is a data engineering job, and it decides answer quality more than the choice of model.

Word matching sometimes wins. On FEVER, a fact-checking task built around named entities, plain BM25 keyword search beat the paper's learned dense retriever. Dense retrieval won on the question answering tasks. That is the origin of the hybrid-search advice you will give on most projects: product codes, names and IDs need keyword matching, paraphrased questions need embeddings.

Retrieval does not guarantee grounding. RAG answered 11.8% of questions correctly even when the right answer was in none of the retrieved passages. The generator was drawing on its own weights. If a customer needs every answer traceable to a source, retrieval alone does not deliver that; you need to check the answer against the passages.