IN THIS BREAKDOWN · 10 SECTIONS
You can read 21% of this breakdown.
The paper in one minute
When a RAG answer is wrong, two things could have failed: retrieval did not bring enough information, or the model did not use what it had. This paper gives a way to tell them apart.
The authors define sufficient context: a question and its context are sufficient if the context supports some plausible answer, one of the right type for the question. The definition deliberately does not need the true answer, so it can be checked at inference time, and a context that confidently states a wrong answer still counts as sufficient. Ambiguity matters: "Ali lives in Paris" is not sufficient for "What country does Ali live in?" until the context also shows which Paris.
They then build an autorater: Gemini 1.5 Pro with instructions and one example labels sufficiency at 93% accuracy on 115 carefully constructed hand-labeled pairs. A smaller model, FLAMe (24B), reaches 87.8% and is used where cost matters. An entailment model that needs the true answer does worse (82.6%).
Stratifying results by sufficiency across FreshQA, HotpotQA and Musique, and across Gemini 1.5 Pro, GPT-4o, Claude 3.5 Sonnet, Gemma 2 27B and Mistral 7B, gives three findings. With sufficient context, models still get a noticeable share wrong. With insufficient context, they mostly answer anyway, wrongly, rather than abstain. And they are surprisingly often right on insufficient context, 35% to 62% of the time, from parametric knowledge, partial clues or lucky guesses.
Finally, selective generation combines the sufficiency label with the model's self-rated confidence in a small logistic regression and abstains below a threshold, improving accuracy among answered questions by 2% to 10%.
Research notes: the problem, approach and results
accuracy of a Gemini 1.5 Pro autorater labeling context sufficiency on 115 hand-labeled pairs, without the true answer
Gemini 1.5 Pro's abstention rate without and with retrieval; retrieved text made it answer far more often
share of insufficient-context questions that strong models still answered correctly
When a RAG system answers wrongly, teams cannot tell whether retrieval failed to bring the needed information or the model failed to use it. Relevance scores and 'does the context contain the answer' checks need a known answer, and do not capture whether the context is enough on its own.
Define sufficient context: the query and context admit a plausible answer, without needing the true answer. Label it with an LLM autorater (Gemini 1.5 Pro with one example, 93% accurate on 115 hand-labeled pairs). Use the label to stratify model behavior, and combine it with the model's self-rated confidence to decide when to answer.
Strong models answered most sufficient-context questions correctly but hallucinated rather than abstained when context was insufficient; adding retrieval cut abstention sharply. Even with insufficient context they were right 35-62% of the time, so abstaining whenever context is insufficient wastes correct answers. Adding the sufficiency signal to confidence improved selective accuracy by 2-10%.
Why it matters to an FDE
When a customer says "the assistant gave a wrong answer", the first question is which half of the system failed. Sufficient context gives you a way to answer it at scale without labeled answers.
Split the error budget. Run a sufficiency autorater over a sample of logged questions and their retrieved context. Wrong answers on sufficient context are generation problems: prompting, model choice, long-context handling. Wrong answers on insufficient context are retrieval problems: chunking, search, missing documents. The fix differs completely.
Measure how often retrieval is enough. On standard benchmarks, only 44.6% to 77.4% of instances had sufficient context at 6,000 tokens. On a customer's corpus, this number tells you how much better retrieval could possibly help.
Expect retrieval to suppress "I don't know". Gemini 1.5 Pro abstained on every question without retrieval in this setup and on 18.6% with it; Claude 3.5 Sonnet went from 84.1% to 52.0%. Retrieved text makes models answer, whether or not it supports the answer.
Don't abstain on insufficiency alone. Because models are right on a large share of insufficient-context questions, refusing whenever context is insufficient throws away many correct answers. Combine the signal with model confidence and pick a threshold for the coverage the customer wants.
The cost is an extra model call per question for the autorater, which is why the paper uses a 24B model for the selective-generation step.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 149 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
The definition's edges, autorater options, findings by model, the method and three misreadings
Free account - 04How it works
Labeling sufficiency, stratifying responses, and building the abstention model
Premium - 05The numbers that matter
Autorater scores, sufficiency by dataset, abstention shifts, selective gains and fine-tuning
Premium - 06Try it yourself
Recheck the paper's tables, then build the selective-generation regression and compare rules
Premium - 07Where it breaks
Small gold set, judge dependence, QA-only scope, cost and internal inconsistencies
Premium - 08Using it on a deployment
Adding a sufficiency check to a customer's RAG pipeline for diagnosis and abstention
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium