The Eval Loop: From Reading Traces to Watching Production
Evaluating an AI agent is a loop, not a score: read real conversations, turn the failures into a dataset, give each failure type a check, validate the judge against human labels, gate releases on repeated runs, and sample live traffic so new failures flow back in. Each step decides what the next one can see, which is why skipping one breaks the rest without raising an error.
TL;DR: An agent's pass rate tells you nothing about which conversations fail or why. The eval loop is six steps that answer that, in order: read traces and name the failures, build a dataset from them, give each failure type a check, prove the judge catches failures, gate releases on repeated runs, and sample production with the validated judge. Failures caught in production go back into the dataset. Every step limits what the next one can see.
Six steps, each feeding the next
Picture a support agent for Calder Equipment, a pump maker. It reads cases, checks warranties and files replacements, and 9% of its conversations go wrong. No dashboard says which 9%. The loop is how you find out, fix it and keep it fixed.
| Step | The question it answers | What it produces | What breaks if you skip it |
|---|---|---|---|
| 1 Read traces | What actually goes wrong? | A list of named failure types | You measure what is easy to measure, not what fails |
| 2 Dataset | Which failures must the next version fix? | Failing inputs with the expected behavior | The fix covers only the cases someone remembered |
| 3 Checks | Which rule catches which failure? | Code checks, a judge, human review, each with a scope | Failure types no rule names pass silently |
| 4 Validate the judge | Does the judge catch real failures? | TPR and TNR against human labels | A judge that says PASS to everything looks accurate |
| 5 CI gate | Does it pass every time, or once? | pass^k on the dataset | A lucky single run ships a flaky agent |
| 6 Production | Did something change after release? | A sampled FAIL rate with an alert | Customers find the regression first |
Play the loop
The widget below is the Calder agent with eight failure types and 90 failures per 1,000 conversations. Pick which types go into the dataset (each one is mostly fixed in v2), switch checks on, then guess how many failures still reach customers before you reveal the answer.
Three results are worth trying for yourself. With no checks, all 90 escape. With all five checks and no dataset work, 15.7 escape, and 7.6 of those are missed safety escalations, because no rule names that failure and only a person reading a 5% sample ever sees one. Code checks alone leave 37.3, and the grounding judge alone leaves 65.1: neither kind of evaluator covers the other's failures.
That last point is the loop's central lesson. A check catches only what its rule describes. Adding more of the same kind of check does nothing for a failure type that none of them describes, and the only way to learn that such a type exists is step 1.
Where teams break the loop
Metrics before reading. A team picks "helpfulness" and "accuracy" because they are standard, then discovers months later that the failures customers complain about are wrong case numbers and claims of success after a tool error. Reading thirty conversations first would have named both.
A dataset built from imagination. Hand-written test cases cover the failures someone thought of. Cases taken from real traces cover the ones that happened.
An unvalidated judge. A judge can agree with human labels 91% of the time and still miss 70% of failures, because most conversations pass. Report its TPR and TNR, not its agreement.
One run in CI. An agent that passes a test 80% of the time passes eight runs in a row only 17% of the time. A single green run hides that.
Nothing watching production. Without a sampled judge, the first signal of a bad deploy is a complaint, which arrives whenever a customer decides to write one.
No way back. A failure found in production and fixed by hand, but never added to the dataset, can return in the next version without anything noticing.
What interviewers are listening for
When asked "how do you know the agent works", strong candidates describe the mechanism in this order: read real conversations, name failure types, build the dataset from them, give each type an evaluator, validate any LLM judge with per-class rates, gate on repeated runs, and sample production. The ordering is the signal. A candidate who starts with a metric has skipped the step that decides which metric matters.
Common misconceptions
- "A high pass rate means the agent is fine." It means most conversations pass. The failures that remain may be the expensive ones.
- "More evaluators means more coverage." Only if they cover different failure types.
- "The judge agrees with us 90% of the time, so it works." Agreement is dominated by the passing majority; look at how many failures it catches.
- "Evals are a launch task." The loop runs for as long as the agent does.
Key takeaways
- The loop's order matters: reading traces decides the dataset, the dataset decides the checks, and the checks decide what production monitoring can catch.
- Each check covers only the failure types its rule names. Map types to checks explicitly and look for the type nobody covers.
- Validate a judge on failures, gate on repeated runs, and feed every production failure back into the dataset.