GO TO A SECTION

↑ ↓ to navigate · Enter to openExplore all sections →
FDEInterviews logo
← RAG & Agent System Design / 63
hardHarveyHebbiaScale

Document extraction scored 95% on the pilot sample and 60% in the first week of production. What happened, and what do you fix first?

Before you touch a model, check whether the two numbers measure the same thing. Field accuracy of 96% across twelve fields is 61% of documents fully correct, which means half of the drop may be arithmetic. The other half is the documents the pilot never saw.

Updated Sep 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.

Before you touch a model, check whether the two numbers measure the same thing. Field accuracy of 96% across twelve fields is 61% of documents fully correct, which means half of the drop may be arithmetic. The other half is the documents the pilot never saw.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 545 remaining answers · ₹2,000 / $25

Every answer, concept and course, every paper breakdown, all published video lessons, hands-on FDE Lab missions, Premium PDF guides and companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.

6 months · One payment · No auto-renewal

Study alongside free video lessons.

Keep practising

Next in this trackYour support agent resolved 69% of conversations last month and 60% this week. Nobody deployed anything. What do you check first?PremiumNext in this trackDesign a simulated-customer regression suite for a support agent in production. What does it catch, what does it miss, how big must it be?PremiumNext in this trackBuild the eval for a text-to-SQL feature over a customer's warehouse: what is in the golden set, what is the metric, how many cases?Premium
FEDITOR'S NOTE

The first thing being scored is whether the candidate asks what 95% and 60% each count before explaining either. Most candidates jump to 'production data is messier', which is true and incomplete. The second thing is the order of fixes: parse before model, and a per-field confidence gate before any of it, because the customer's real question is not 'how accurate is it' but 'which fields can I stop checking'.

DISCUSSION · 0

No comments yet — be the first to share your approach.