Read the transcripts before you pick a metric
Calder wants a quality number for its support agent. Predict how many of six conversations fail, label them against the policy, mark the first turn that broke, and find the failure type the supplied automated checks miss.
- beginner
- about 20 min
- 6 objectives
- par 7 steps
- Calder Equipment
Label conversations against a written policy, not the customer's mood
Read the current task at the top of the console. Inspect evidence with the terminal or Files tab, then use the answer field to submit your findings. Tasks that ask for a fix check your file or service changes automatically.
No installation needed. The terminal runs supported commands and recorded programs in a simulated system. Need help? offers a starting point, then a clue or a worked example. Take time to investigate; the debrief includes a question about applying the lesson elsewhere. Your run is saved in this browser, and completed runs can be verified and saved to your account.
- Customer
- Calder Equipment (synthetic customer from the Production AI Agents course; every conversation on this box is synthetic, written to teach error analysis)
- Situation
- The release team wants one quality number for the support agent by the end of the week, and someone has suggested pointing an LLM judge at the whole log. Nobody has read the conversations yet. Six from last week are on this box, with the policy the agent is meant to follow.
- Your role
- You are the FDE asked to start the evaluation. You have the transcripts, the policy, a labels file and a checker that shows the reviewer key once you have committed.
- Goal
- Predict first, then label each conversation pass or fail, mark the first turn that went wrong, type each failure, and name the type that escapes the supplied shape and grounding checks.
- 1.Predict how many of the six conversations fail, before reading them (you commit to an answer)
- 2.Label all six conversations pass or fail against the policy
- 3.Mark the first turn that went wrong in each failure
- 4.Give each failure a type a second reviewer could apply
- 5.Compare your labels and your prediction with the reviewer key
- 6.Name the failure type the supplied checks miss (you commit to an answer)