The candidate that scores higher
Calder's new agent release scores 10 of 11 against the old one's 8. Read the paired results case by case, find the case where the better score hides a worse action, and say what should happen to this release.
- beginner
- about 20 min
- 4 objectives
- par 8 steps
- Calder Equipment
Read a paired release comparison one case at a time
Read the current task at the top of the console. Inspect evidence with the terminal or Files tab, then use the answer field to submit your findings. Tasks that ask for a fix check your file or service changes automatically.
No installation needed. The terminal runs supported commands and recorded programs in a simulated system. Need help? offers a starting point, then a clue or a worked example. Take time to investigate; the debrief includes a question about applying the lesson elsewhere. Your run is saved in this browser, and completed runs can be verified and saved to your account.
- Customer
- Calder Equipment (synthetic customer from the Production AI Agents course; the release packet is a synthetic record written to teach evaluation, not the output of any model)
- Situation
- Two releases of the support agent ran the same eleven recorded cases. The candidate scored 10 of 11; the baseline 8. The product lead is pleased. The evaluation owner is on leave and left a paired packet and a one-line note:
read it case by case. - Your role
- You are the FDE asked to sign off. You have the packet,
jq, and the console. Nothing here is a real model's output; it is the shape of a real release comparison. - Goal
- List which cases improved and which got worse, find the case whose score hides an effect nobody approved, and say whether this release ships.
- 1.Count the cases that went from failure to success (you commit to an answer)
- 2.Find the case whose candidate run produced an effect nobody approved (you commit to an answer)
- 3.Find the case with no recorded outcome (you commit to an answer)
- 4.Say what happens to this release (you commit to an answer)