Challenge · Evaluation and release · Premium
Golden set drift
An eval passes while the assistant contradicts the current coverage policy. Find stale golden labels, update their policy references, version the manifest, and rerun against unchanged model answers.
- intermediate
- about 25 min
- 5 objectives
- par 10 steps
- Meridian Health
Mission workspace
This mission is part of premium, with every challenge and incident in the lab. The brief below is the whole problem; the box, the hints and the debrief are what you unlock.
Get full Premium access · ₹2,000 / $25
Every answer, concept and course, every paper breakdown, all published video lessons, hands-on FDE Lab missions, Premium PDF guides and companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.
6 months · One payment · No auto-renewal
Study alongside free video lessons.
Brief
- Customer
- Meridian Health (synthetic customer; every case, label and policy on this box is made up for the exercise)
- Situation
- Coverage policy CP-7 changed on April 1: dental now requires a referral. The nightly evaluation still passes 5 of 5. Supervisors report the assistant tells members no referral is needed. The golden set's expected answers say the same thing the assistant does, because they were written in January.
- Your role
- You are the FDE who owns the evaluation set. You have the golden set, the policy history, the manifest, and the runner. You may edit expected answers only with the policy reference that justifies each change.
- Goal
- Find the cases whose expected answers contradict the current policy, fix them with a reference, version the manifest so the run names the policy it was scored against, and show the run failing honestly.
Objectives
- 1.Count the golden cases whose expected answer contradicts the current policy (you commit to an answer)
- 2.Name why a passing evaluation missed it (you commit to an answer)
- 3.Copy the policy’s canonical reference answers, update the reviewed references, and version the manifest
- 4.Predict the score once the labels match the current policy (you commit to an answer)
- 5.Show the run failing honestly against the named policy