TL;DR: Build a 50 to 100 case golden set from real user questions labeled with source passages, measure retrieval and generation on separate metrics so a launch gate can fail one without masking the other, then freeze that gate as your CI regression suite.
How to approach it
The move that sounds rigorous and fails is a single number from a judge nobody calibrated: "87% on our test set." Ask who wrote the set and how the judge was checked, and the number usually dissolves in the room. This is the question behind every other design question: at AI labs, raising evals unprompted is the strongest single signal in the loop. Structure your answer in three layers: build a golden set, measure retrieval and generation separately, then define a launch gate with the customer.
A strong answer
Start with the golden set: 50–100 questions sourced from real users (pilot logs, support tickets, SME interviews), not invented by the dev team, because invented questions are suspiciously well-phrased and miss the jargon, typos, and ambiguity of real traffic. For each: the question, the ground-truth answer, and the source document/passage. Include adversarial slices: questions whose answer is not in the corpus (the system should refuse), ambiguous questions, multi-document questions, and stale-data traps.
One labeled case makes the artifact concrete, because "golden set" stays hand-wavy until you have seen a row of one:
question: "wat is teh SLA if lakehouse ingest is delayed?" <- real ticket, typo and all
truth: "4 business hours for P1 sources, next day otherwise (Ops Runbook 3.2)"
source: ops-runbook.pdf, section 3.2, chunk id 8841
slice: answerable / single-doc
---
question: "can we terminate the Danube contract early?"
truth: REFUSE, that contract is not in the corpus
source: none
slice: not-in-corpus (adversarial)
The second case is the kind dev-authored sets never contain, and it is where enterprise launches actually fail. Keep the typo in the first one; cleaning it up quietly optimizes your system for traffic that does not exist.
Then measure in two stages, because a blended "accuracy" number can't tell you what to fix. Retrieval: recall@k and MRR against labeled source passages, did the right chunk make it into the context? Generation: given the right context, is the answer faithful (no claims unsupported by context) and correct? Faithfulness and correctness can be graded by an LLM-as-judge with a tight rubric, spot-calibrated against ~50 human labels (aim for >85–90% agreement before trusting it). That target has a reference point: Zheng et al. (2023, 'Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena') found strong judge models agree with human preferences over 80% of the time, the same level at which two humans agree with each other. Your domain rubric is narrower than open chat, so expect to do a little better, and treat anything below the human-to-human line as a rubric problem rather than a model problem. Calibration is mechanical, not mystical: have the judge and a domain expert grade the same 50 cases independently, count agreements, and look hardest at the disagreements, because they cluster (judges are systematically lenient on fluent-but-unsupported answers, which is exactly the failure you built the eval to catch). If agreement is low, tighten the rubric and re-run; if it stays low, the dimension is too subjective to automate and a human stays in that loop. This split is diagnostic gold: 60% end-to-end accuracy decomposes into "retrieval recall is 70%, generation is right 85% of the time when retrieval succeeds", so now you know to fix retrieval.
The flowchart's two arrows into the launch gate are the point. Retrieval and generation reach the gate on separate wires, so the gate can fail one while the other passes, and you know which fix to fund before anyone argues about it.
Also measure the refusal behavior: false-answer rate on the not-in-corpus slice is often the launch-blocking metric for enterprise customers, since one confident hallucination costs more trust than ten "I don't know"s.
Finally, the launch gate, negotiated with the customer: for example, ≥85% correct on the golden set, ≤2% confident-wrong on the adversarial slice, p95 latency under 5s, and a defined escalation path for failures. Write it down; it converts "the demo felt good" into an explicit, re-runnable contract, and the same suite becomes your CI regression gate for every prompt or model change after launch.
The two stages and what a low score points to:
| Stage | Metric | What a low score means |
|---|---|---|
| Retrieval | recall@k, MRR | The right chunk did not make it into the context |
| Generation | Faithfulness, correctness | Given the right context, the answer adds unsupported claims or is wrong |
Note: the stages multiply, so 70% retrieval recall and 85% generation correctness decompose a 60% end-to-end number, telling you to fix retrieval first.
What interviewers probe next
"How big does the golden set need to be?" 50 cases detect large regressions; 200+ for finer slices; grow it from production failures weekly. "Can you trust LLM-as-judge?" For faithfulness with a rubric and calibration, yes; for open-ended quality, weakly. "What about post-launch?" Sample live traffic for grading, add thumbs-down cases to the set, watch retrieval-score distributions for drift.
What gets proposed in the room, and the version that survives a launch:
| What people reach for | Why it fails | What to say instead |
|---|---|---|
| One blended "accuracy" score | Cannot say whether retrieval or generation failed, so you fix the wrong stage | "recall@k for retrieval; faithfulness and correctness for generation" |
| Test questions written by the dev team | Too well phrased; none of the typos, jargon or ambiguity of real traffic | "Real tickets and pilot logs, labeled by domain experts" |
| An LLM judge nobody calibrated | Systematically lenient on fluent, unsupported answers, the exact failure the eval exists to catch | "Fifty human-labeled cases, agreement counted, disagreements read" |
| No refusal slice | One confident wrong answer costs more trust than ten honest 'I don't know's | "Not-in-corpus questions with REFUSE as the ground truth" |
Common mistakes
Saying "we'd test it" without naming a metric, a set size, or who writes the questions. Reporting one blended accuracy number that can't localize failures. Letting developers author all the eval cases. And skipping the refusal slice: interviewers playing enterprise customers will ask "what happens when the answer isn't in the docs?" and grade your silence.
What to actually do
Pull 50 to 100 questions from real traffic and have domain experts label the answer and the source passage for each. Add a not-in-corpus slice with REFUSE as the truth. Compute recall@k and MRR on retrieval, faithfulness and correctness on generation, and never blend them into one number. Grade 50 cases with both a human and the judge, count the agreement, and read every disagreement. Write the launch gate down with the customer, thresholds and a p95 included, then run the same suite in CI on every prompt and model change.
Key takeaways
- Source the golden set from real users and SMEs, not the dev team, and label each case with its source passage.
- Split retrieval from generation metrics so the gate localizes failure instead of hiding it in one blended score.
- Calibrate any LLM-as-judge against ~50 human labels, and freeze the launch gate as your CI regression suite.