TL;DR: Resist the urge to propose fixes. Build an eval set from failing traffic and split each failure down a tree (not in corpus, not retrieved, retrieved-but-ignored, wrongly answered, mislabeled), then quantify the buckets so prioritization becomes arithmetic.
How to approach it
The trap is proposing fixes ("try a reranker! better chunking!") before measuring anything. The first sentence interviewers want to hear: "Before changing anything, I'd build an eval set from the failing traffic so we can see where the 60% breaks." Tuning without diagnosis is how pilots die. You fix retrieval when the problem was ingestion, burn two weeks, and lose the customer's confidence.
A strong answer
Step 1: Get the failures. Pull the pilot's logs: queries, retrieved chunks, answers. Label 50–100 cases with what the right answer was and where it lives. If logs weren't kept, that's finding #1; instrument first.
Step 2: Split every failure down a tree, because each branch has a different fix:
- Answer not in the corpus at all: ingestion/coverage problem: missing connector, failed PDF parse, stale sync. Fix: ingestion audit. (Embarrassingly common, often 10–20% of "model is dumb" complaints.)
- In corpus, not retrieved (not in top-k): retrieval miss: vocabulary mismatch, bad chunk boundaries, wrong k, missing hybrid/BM25 leg. Fix: hybrid search, reranker, chunking, or query rewriting, chosen by why it missed.
- Retrieved, but model ignored or contradicted it: generation/faithfulness: prompt doesn't enforce grounding, context too long (lost-in-the-middle), conflicting chunks. Fix: prompt, context assembly, chunk ordering.
- Answered when it shouldn't have (no good source): missing refusal behavior. Fix: grounding instructions, an "I don't know" path, and a threshold on retrieval scores.
- "Wrong" but actually right: expectation mismatch where the label or the user's premise is wrong. Fix: conversation with the customer about ground truth.
Step 3: Quantify the split, e.g. "of 80 failures: 15% coverage, 50% retrieval, 25% generation, 10% mislabeled." Now prioritization is arithmetic: retrieval is the biggest bucket, so hybrid + reranker first, and the predicted ceiling if retrieval were perfect is visible from the data.
The quantified split also hands you the sentence that wins the customer meeting, because each bucket converts to accuracy points with one multiplication. Failures are 40% of traffic, so a bucket that is 50% of failures is worth at most 0.5 x 40 = 20 points of accuracy; the 15% ingestion bucket is worth 6 points; the 25% generation bucket, 10; mislabels, 4. Now you can say "a perfect retrieval fix takes us from 60% to at most 80%, and the demo-week plan is the retrieval bucket plus the ingestion audit, ceiling 86%", which is a fundamentally different conversation from "we will try a reranker and see." The word ceiling is doing real work: it sets expectations honestly (a reranker will not capture the full 20 points), and it tells everyone in advance what success looks like, so week two is measured against a number both sides agreed to rather than a feeling.
Step 4: Fix, re-run the eval, repeat. Each change is re-scored against the same set, so improvement is a number the customer can see (60% to 74% to 85%), and the eval becomes the regression gate for everything after.
Also say the meta-thing: 60% with no eval infrastructure is a process problem. The deliverable isn't a smarter pipeline; it's a measurement loop the customer can keep running.
What interviewers probe next
"Suppose 70% of failures are retrieval, then what?" (Subdivide: vocabulary vs chunking vs ranking, and pick the technique that matches.) "Who labels ground truth?" (Customer SMEs, and disagreements between SMEs are themselves a finding.) "The customer wants it fixed this week." (Ship the top-bucket fix behind the eval; promise the number, not perfection.)
Common mistakes
Listing fixes as a buffet with no diagnostic order, the single most-cited rejection reason on this question. No failure taxonomy. Not asking what "correct" means or who judged the 60%. And forgetting refusals: some of the missing 40% should become honest "I don't know"s, not forced answers.
Key takeaways
- Measure before you fix; the first move is an eval set built from failing traffic, not a reranker.
- The retrieved-or-not split routes you to a retrieval bug versus a generation bug and saves you tuning the wrong half.
- Quantifying the failure buckets turns prioritization into arithmetic and gives the customer a visible number each iteration.