Challenge · Evaluation and release · Premium
The judge that agrees 94 percent
Calder's grounding judge agrees with the reviewer on 94% of 200 conversations. Split agreement into failures caught and passes passed, find why the judge misses half the failures, fix it, and correct the dashboard's failure rate.
- advanced
- about 35 min
- 7 objectives
- par 12 steps
- Calder Equipment
Mission workspace
This mission is part of premium, with every challenge and incident in the lab. The brief below is the whole problem; the box, the hints and the debrief are what you unlock.
Get full Premium access · ₹2,000 / $25
Every answer, concept and course, every paper breakdown, all published video lessons, hands-on FDE Lab missions, Premium PDF guides and companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.
6 months · One payment · No auto-renewal
Study alongside free video lessons.
Brief
- Customer
- Calder Equipment (synthetic customer from the Production AI Agents course; every label, count and rate on this box is synthetic and made up for the exercise)
- Situation
- A model judge checks one criterion: does the reply state a date, price or status that no tool returned? A reviewer labeled 200 random conversations and the judge agreed on 188 of them. The weekly dashboard reports the judge's raw FAIL share as the grounding failure rate: 5.4% last week.
- Your role
- You own the judge. You have the natural-mix labels, a failure-heavy labeled set, the judge config, and recorded runs of the judge over both.
- Goal
- Measure what 94% hides, find and fix why the judge misses failures, validate the fix on enough failures to trust it, and replace the dashboard's raw number with a corrected estimate and its interval.
Objectives
- 1.Find the agreement a judge that always says PASS would get (you commit to an answer)
- 2.Count the reviewer's failures the judge also failed (you commit to an answer)
- 3.Measure the judge on enough failures to trust its TPR
- 4.Name why the judge misses failures (you commit to an answer)
- 5.Give the judge the evidence and validate it again
- 6.Predict the true grounding failure rate before correcting the dashboard (you commit to an answer)
- 7.Replace the dashboard number with a corrected estimate and its interval