GO TO A SECTION

↑ ↓ to navigate · Enter to openExplore all sections →
FDEInterviews logo
← RAG & Agent System Design / 66
hardDatabricksSnowflakeGlean

Build the eval for a text-to-SQL feature over a customer's warehouse: what is in the golden set, what is the metric, how many cases?

Three correct queries scored 0 of 3 on exact match and 3 of 3 on execution. The golden set is question, reference SQL and result set, stratified by the ways queries go wrong, with abstention as its own column. A hundred cases gives plus or minus eight points; four hundred gives four.

Updated Sep 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.

Three correct queries scored 0 of 3 on exact match and 3 of 3 on execution. The golden set is question, reference SQL and result set, stratified by the ways queries go wrong, with abstention as its own column. A hundred cases gives plus or minus eight points; four hundred gives four.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 545 remaining answers · ₹2,000 / $25

Every answer, concept and course, every paper breakdown, all published video lessons, hands-on FDE Lab missions, Premium PDF guides and companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.

6 months · One payment · No auto-renewal

Study alongside free video lessons.

Keep practising

Next in this trackA chunking change shipped last week. Aggregate recall barely moved, and support says answers got worse. Find the regression.PremiumNext in this trackRetrieval metrics are green, the correct passage is provably in the context, and the answer is still wrong. Where do you look?PremiumNext in this trackAfter an index rebuild, a set of documents stopped appearing in any search result. Nothing errored and the count is right. Find them.Premium
FEDITOR'S NOTE

Scored on whether the candidate can say what the metric compares and why, and whether they know the size arithmetic. Strong answers build the set from questions people actually asked, including the ones that should be refused, and report per stratum. Weak answers propose exact match, or a single accuracy over a set of easy single-table questions the pilot happened to include.

DISCUSSION · 0

No comments yet — be the first to share your approach.