GO TO A SECTION

↑ ↓ to navigate · Enter to openExplore all sections →
FDEInterviews logo
← RAG & Agent System Design / 67
hardPalantirGleanHebbia

A chunking change shipped last week. Aggregate recall barely moved, and support says answers got worse. Find the regression.

The eval set says recall fell 2.3 points, which is inside the noise you would expect from a re-index. Support says the product got worse. Both are right, and the reason is that one document class lost 49 points of recall while everything else improved slightly.

Updated Sep 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.

The eval set says recall fell 2.3 points, which is inside the noise you would expect from a re-index. Support says the product got worse. Both are right, and the reason is that one document class lost 49 points of recall while everything else improved slightly.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 545 remaining answers · ₹2,000 / $25

Every answer, concept and course, every paper breakdown, all published video lessons, hands-on FDE Lab missions, Premium PDF guides and companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.

6 months · One payment · No auto-renewal

Study alongside free video lessons.

Keep practising

Next in this trackRetrieval metrics are green, the correct passage is provably in the context, and the answer is still wrong. Where do you look?PremiumNext in this trackAfter an index rebuild, a set of documents stopped appearing in any search result. Nothing errored and the count is right. Find them.PremiumNext in this trackYou added a cross-encoder reranker. p95 latency roughly doubled and answer quality did not move. Explain what happened and fix it.Premium
FEDITOR'S NOTE

This is a statistics question hiding in a retrieval costume, and the candidates who do well say so early. The aggregate is a weighted average, so a segment worth 8% of traffic can lose half its recall and move the headline by two points. The number worth carrying into the interview is the power calculation: the drop needs about eleven thousand queries per arm to detect in aggregate and sixteen inside the right stratum. Anyone who reaches for a bigger eval set rather than a partitioned one is solving it the expensive way.

DISCUSSION · 0

No comments yet — be the first to share your approach.