Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

When Gemini 1.5 accepted a million tokens and GPT-4o 128,000, a real question opened up: why build retrieval at all? Earlier comparisons had found retrieval winning, but they used weaker models and shorter contexts.

Google DeepMind and the University of Michigan reran the comparison with Gemini-1.5-Pro, GPT-4o and GPT-3.5-Turbo on nine long-document tasks from LongBench and ∞Bench: question answering over stories, papers and Wikipedia, multi-hop questions, and meeting summarization. Long context (LC) meant sending the whole document. RAG meant splitting it into 300-word chunks and sending the five most similar to the question.

Long context won on quality, by 7.6 points on average for Gemini, 13.1 for GPT-4o and 3.6 for GPT-3.5-Turbo. But two other observations shaped the paper. RAG costs a fraction of the tokens. And for 63% of queries the two methods produced exactly the same answer, right or wrong.

So the authors proposed Self-Route. First, answer with RAG, but let the model reply "unanswerable" if the retrieved chunks are not enough. Only for those queries, send the full context. Most queries stopped at the first step, and the overall score landed close to long context at well under the cost: 38% of the tokens for Gemini, 61% for GPT-4o, with GPT-3.5-Turbo actually scoring higher than either method alone.

"Should we just put the whole document in the prompt?" comes up on every document-heavy project. This paper gives you a measured answer to bring to the conversation.

Quality and cost pull in opposite directions. Long context was better on average with strong models. RAG was dramatically cheaper because pricing follows input tokens. Neither is "right"; the choice is a budget decision.

You do not have to choose once. The routing idea is simple to build: give the model the retrieved chunks and permission to say it cannot answer. Escalate only those queries. On the paper's tasks, 57% to 77% of queries never needed the long call. That is the kind of design that makes a pilot's unit economics work in production.

Retrieval is fragile to wording. The most striking result is small. On a test where a passkey is hidden in long filler text, RAG found it 80% of the time when asked "What is the passkey?" Reword the question as "What is the special token hidden inside the texts?" and RAG dropped to under 5%, while long context barely moved. Retrieval depends on the question sharing words or meaning with the passage. Users rarely phrase questions the way documents are written.