GO TO A SECTION

↑ ↓ to navigate · Enter to openExplore all sections →
FDEInterviews logo
← RAG & Agent System Design / 65
hardSierraDecagonAnthropic

Design a simulated-customer regression suite for a support agent in production. What does it catch, what does it miss, how big must it be?

A nightly suite of fifty simulated conversations per intent catches a collapsed tool with certainty and a five-point drift almost never. Knowing which is which, and sizing each layer for what it can actually see, is the whole design. The grader checks outcomes, not transcripts, or the agent learns to sound resolved.

Updated Sep 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.

A nightly suite of fifty simulated conversations per intent catches a collapsed tool with certainty and a five-point drift almost never. Knowing which is which, and sizing each layer for what it can actually see, is the whole design. The grader checks outcomes, not transcripts, or the agent learns to sound resolved.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 545 remaining answers · ₹2,000 / $25

Every answer, concept and course, every paper breakdown, all published video lessons, hands-on FDE Lab missions, Premium PDF guides and companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.

6 months · One payment · No auto-renewal

Study alongside free video lessons.

Keep practising

Next in this trackBuild the eval for a text-to-SQL feature over a customer's warehouse: what is in the golden set, what is the metric, how many cases?PremiumNext in this trackA chunking change shipped last week. Aggregate recall barely moved, and support says answers got worse. Find the regression.PremiumNext in this trackRetrieval metrics are green, the correct passage is provably in the context, and the answer is still wrong. Where do you look?Premium
FEDITOR'S NOTE

This is the follow-on to the live-metric triage question and it is asked at the same companies. The strong candidate separates two jobs the suite is asked to do (catch a break overnight, measure slow drift) and gives them different instruments with the arithmetic that says why. The weak candidate proposes 'run a thousand simulated conversations' with no grader design and no statement of what the number means.

DISCUSSION · 0

No comments yet — be the first to share your approach.