Tune the production judge
Calder's live grounding judge pages so often that on-call muted it. Backtest the monitor on a recorded week, tune sample share, threshold and window to meet three constraints, then show the setting holds on weeks you did not tune on.
- advanced
- about 35 min
- 5 objectives
- par 11 steps
- Calder Equipment
This mission is part of premium, with every challenge and incident in the lab. The brief below is the whole problem; the box, the hints and the debrief are what you unlock.
Every answer, concept and course, every paper breakdown, all published video lessons, hands-on FDE Lab missions, Premium PDF guides and companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.
6 months · One payment · No auto-renewal
Study alongside free video lessons.
- Customer
- Calder Equipment (synthetic customer from the Production AI Agents course; every count and rate on this box comes from a seeded simulation made up for the exercise)
- Situation
- A validated grounding judge (TPR 0.88, TNR 0.97) scores a sample of live support conversations and pages when the FAIL rate over a window crosses a threshold. It pages on ordinary hours, the channel is muted, and a prompt change that raised true failures from 5% to 9% was found a day late by the human review queue.
- Your role
- You own the monitor. You can edit monitor.yaml and backtest it against the recorded week, or replay it across fifty other recorded weeks.
- Goal
- Meet all three constraints in CONSTRAINTS.md on the recorded week, explain why the starting config pages, and show your setting holds on at least 35 of 50 other weeks.
- 1.Predict the false alerts the starting config raises in a normal week (you commit to an answer)
- 2.Name why the starting config pages on normal hours (you commit to an answer)
- 3.Meet all three constraints on the recorded week
- 4.Name the lever that buys precision with detection time instead of judge calls (you commit to an answer)
- 5.Show your setting holds on at least 35 of 50 other weeks