Retrieval-Augmented Generation (RAG)

TL;DR: Retrieval-Augmented Generation answers a question by first retrieving the most relevant chunks of your data and then generating an answer from them, grounded in that retrieved context. It beats fine-tuning for knowledge that changes or must be cited, and it fails in two distinct places, retrieval and generation, which you must evaluate separately.

The idea in one picture

A raw language model answers from what it absorbed during training: frozen, undated, and impossible to cite. RAG fixes that by giving the model fresh, specific context at the moment you ask. You keep your knowledge in an external store, find the passages that match the question, and hand those passages to the model as part of the prompt. The model then writes an answer constrained by what you retrieved.

In the flowchart the dotted arrow is the part that runs once, and the solid path is what runs on every question. Everything to the left of "Build prompt" is retrieval; everything to the right is generation, and any bug lives on exactly one side.

Two terms in the boxes: a chunk is a passage of a few hundred tokens that a document was split into when it was indexed, and top-k means the k chunks whose vectors sit nearest the question's, typically three to ten.

The name comes from Lewis et al. (2020, 'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks'), who paired a pre-trained generator, the parametric memory, with a dense vector index of Wikipedia as non-parametric memory, and found the combination produced more specific and factual language than the generator alone, with a provenance trail the parametric model could not give. Every production RAG system is that split with your documents in place of Wikipedia.

The left half (embed, retrieve) is an information-retrieval problem. The right half (prompt, generate) is a language problem. Holding those two halves apart is the single most important habit when you build or debug RAG.

A worked example

Suppose a customer asks "what is our refund window for enterprise plans?" A frozen model might guess "30 days" because that is common. RAG instead retrieves the actual policy paragraph and answers from it. Suppose, hypothetically, the retrieved chunk reads "Enterprise plan refunds are accepted within 45 days of invoice." The answer now says 45 days and cites that paragraph. Without retrieval the model's 30 was a guess with the confidence of a fact.

def answer(question, store, llm, k=5):
    q_vec = embed(question)                 # same model used to index
    chunks = store.search(q_vec, k=k)       # nearest neighbors by cosine
    context = "\n\n".join(c.text for c in chunks)
    prompt = (
        "Answer using ONLY the context. If it is not there, say you do not know.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    return llm.generate(prompt), chunks      # return sources for citation

Two design choices already matter here: the instruction to refuse when the context lacks the answer (this is most of your hallucination defense), and returning the chunks so the answer can cite them.

Why interviewers probe this

The answer that loses is "with RAG the model can't hallucinate any more." Retrieval can miss, the model can ignore what it was handed, and a candidate who cannot say which of those happened cannot debug the system they just drew. RAG is the modal enterprise deployment, so the loop wants to see that you understand the trade-offs, not just the diagram. The classic question is "fine-tune, RAG, or prompt?" The honest answer: reach for RAG when the knowledge changes often, must be cited, or is too large to bake into weights; reach for fine-tuning when you need to change the model's behavior or format rather than its facts. They are complementary, not rivals.

The second probe is always evaluation. A strong candidate says "I measure retrieval and generation separately": retrieval with recall@k and precision@k against a golden set (recall@k: of the questions whose needed passage exists in the corpus, how often it appears among the k retrieved; precision@k: of the k retrieved, how many are relevant), generation with faithfulness (does every claim in the answer follow from the context) and answer relevance (does it answer the question that was asked). If recall@k is low, no prompt engineering will save you, so you fix retrieval first.

The decision the loop is really asking about:

Common misconceptions

• "RAG eliminates hallucination." It reduces it by grounding, but the model can still ignore or misread the context. You need the refuse-when-unsupported instruction and a faithfulness metric.
• "Just stuff more documents in the prompt." Irrelevant context degrades the answer and burns tokens. Precision matters as much as recall; this is why reranking exists.
• "Bigger chunks are better." Chunking is a real tuning decision. Chunks too large dilute relevance, too small lose the context a passage needs to make sense.
• "Retrieval quality is the model's job." It is mostly your embedding, chunking, and indexing choices. The generator can only work with what you hand it.

Key takeaways

• RAG = retrieve relevant chunks, then generate an answer grounded in them; it keeps knowledge fresh and citable without retraining.
• It fails in two separable places: retrieval and generation. Evaluate and fix them independently.
• Choose RAG for changing or citable knowledge, fine-tuning for changed behavior or format; they compose.
• Most of your accuracy comes from retrieval quality (embeddings, chunking, indexing) and a prompt that refuses when the context does not support an answer.
• Decide the refusal instruction and the citation format before the first document is indexed; both are cheap to build in and expensive to retrofit.