Challenge · intermediate · 30 min · Premium

The regression gate

Calder's candidate release scores higher than the baseline and someone wants it shipped today. Read the paired packet, find the case that should block it, turn that failure into a release-blocking test, and make the gate say hold.

Brief

Customer: Calder Equipment (synthetic customer from the Production AI Agents course; the release packet is a synthetic record written to teach evaluation, not the output of any model)
Situation: The evaluation run for the support agent's candidate release is in. Success went from 8 of 11 to 9 of 11 and the product lead has asked for a same-day release. The current gate in gate.yaml checks one thing: that success did not go down. The paired packet holds more than that number, and the person who wrote the gate has moved teams.
Your role: You are the FDE who signs the release review. You may read the packet, run the evaluation CLI (evalctl) over it, and edit gate.yaml and the tests in tests/. The packet is an observation of a recorded run; editing it makes a counterfactual, not a result, and the CLI knows the difference.
Goal: Name the case that must block this release and the case whose outcome is missing, make the gate block on unauthorized effects and missing outcomes, get a verdict of hold with reasons from the unedited packet, and add the test that would have caught the regression before anyone looked.

Objectives

1.Name the case whose candidate run produced an effect without an approval (you commit to an answer)
2.Name the case with no recorded outcome (you commit to an answer)
3.Make the gate block on unauthorized effects and on missing outcomes
4.Get a verdict of hold, with both reasons, from the unedited packet
5.Add the regression test that would have blocked this candidate before review

You will practise

→Read a paired release comparison case by case instead of by the headline number
→Distinguish a safety regression from a success regression and rank it above any gain
→Treat a missing outcome as blocking, not as neutral
→Turn a diagnosed failure into a release-blocking regression test

The mechanism, explained

Course lesson: read an agent evaluation without losing the failures
the paired-reading method this mission drills, on the same packet shape
Course lab: choose the next release from paired evidence
the worked version of C07, C08, C10 and C11; do the mission first, then compare
Regression suites and CI gates
why a diagnosed failure becomes a permanent check rather than a note
Agentic evals and trajectories
outcomes, effects and paths: what an agent evaluation has to record to be readable
Design an agent evaluation harness, offline and online
the interview question whose answer is the gate you are about to write

Mission workspace

This mission is part of premium, with every challenge and incident in the lab. The brief above is the whole problem; the box, the hints and the debrief are what you unlock.

Get full Premium access · ₹2,000 / $25

Every answer, concept and course, plus Premium PDF guides, companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.

6 months · One payment · No auto-renewal

Study alongside free video lessons.

← all missions next: Does it fit →