Challenge · Evaluation and release · Premium
Passes once, fails at eight
Calder's candidate passed 18 of 20 tasks in CI and the gate says ship. Rerun every task eight times, find the task that passes most runs and files an unconfirmed replacement on the rest, read why, and make the gate hold.
- intermediate
- about 30 min
- 7 objectives
- par 12 steps
- Calder Equipment
Mission workspace
This mission is part of premium, with every challenge and incident in the lab. The brief below is the whole problem; the box, the hints and the debrief are what you unlock.
Get full Premium access · ₹2,000 / $25
Every answer, concept and course, every paper breakdown, all published video lessons, hands-on FDE Lab missions, Premium PDF guides and companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.
6 months · One payment · No auto-renewal
Study alongside free video lessons.
Brief
- Customer
- Calder Equipment (synthetic customer from the Production AI Agents course; every task, run and transcript on this box is synthetic and made up for the exercise)
- Situation
- The release gate runs each of 20 tasks once and ships when the pass rate is at least 0.85. Candidate v14 passed 18 of 20. Graders are deterministic trace rules, so a task that passes and fails across runs is the agent varying, not the grader.
- Your role
- You own the release gate. You can rerun the suite with repeats, read the transcripts it records, and edit gate.yaml.
- Goal
- Measure how many tasks pass every run, find the flaky task that writes, name why it fails, and change the gate so this candidate is held for the right reasons.
Objectives
- 1.Predict how many tasks pass all eight runs (you commit to an answer)
- 2.Rerun every task eight times
- 3.Count the tasks that pass some runs and fail others (you commit to an answer)
- 4.Name the flaky task that can do irreversible harm (you commit to an answer)
- 5.Name why T07 fails on some runs (you commit to an answer)
- 6.Make the gate run eight times, require every run, and hold write tasks to every run
- 7.Get a hold verdict with both reasons from the unedited results