Error Analysis: Read the Traces Before Choosing a Metric
Error analysis is reading real conversations, marking each pass or fail, writing down where each failure first went wrong, and grouping those notes into failure types you can count. It comes before any metric or judge, because it is the only step that tells you which failures exist, and every later eval decision depends on that list.
TL;DR: Before you pick a metric, read conversations. Mark each one pass or fail, write one short note at the first turn where a failure went wrong, then group the notes into named failure types and count them. That list decides your dataset, your checks and your judge prompts. Skipping it is how teams end up measuring "helpfulness" while customers suffer from wrong case numbers.
Why reading comes first
An LLM system fails in ways nobody predicted on the whiteboard. A support agent can invent a delivery date, file a replacement on an expired warranty, report success after a tool timed out, or book a routine visit for a customer who just said the pump is sparking. None of those show up in an aggregate score. All of them show up the moment someone reads the conversation.
A metric chosen before reading measures what was easy to imagine. Error analysis finds out what actually happens, and only then chooses what to measure.
The procedure
Sample. Take a random sample so the counts mean something, and add targeted cases (thumbs-down, escalations, long conversations) so rare failures appear at all. Keep the two apart when you count, because the targeted ones over-represent failure.
Label and note. For each conversation, decide pass or fail against the product's actual policy, then write one plain sentence at the first turn where it went wrong. Qualitative researchers call this open coding. Note the first error, not the last: a wrong case lookup in turn two makes everything after it wrong, and fixing turn five fixes nothing.
Group. Sort the notes into types that a person could check consistently (axial coding). "Bad answer" is not a type. "Stated a delivery date no tool returned" is.
Count and rank. Frequency alone misleads. A rare failure that ships a pump to the wrong site outranks a common one that phrases a greeting oddly.
Stop when types stop appearing. Keep reading until new conversations only add to existing types. That is the saturation point, and it is a better stopping rule than a fixed number.
Try it
Six synthetic Calder conversations follow. Label each, click the turn where it broke, and see the truth and which kind of check would catch it. Two of them pass, and one of those involves an unhappy customer.
The unhappy customer is the case people get wrong. The agent declined an out-of-warranty refund, stated the reason and handed off to sales, which is exactly what the policy says to do. Marking it FAIL is a false alarm, and false alarms are not harmless: they fill the dataset with cases the next version should not change.
The sparking pump is the other lesson. No code check or grounding judge looks for a missed safety escalation, so the only way that type ever enters your list is a person reading the conversation.
From notes to the rest of the loop
| Note (open code) | Failure type | What catches it |
|---|---|---|
| "Said Thursday; shipment tool had no date" | Ungrounded claim | LLM judge with a grounding rule |
| "Filed warranty replacement, warranty expired in March" | Replacement outside policy | Code check on the final state |
| "Tool timed out, told customer it shipped" | Success claimed after a tool error | Trace rule: error followed by a success claim |
| "Customer reported sparking, agent booked next week" | Missed safety escalation | A new escalation rule, or human review |
Add a count column from your own sample; the counts belong to your traces, not to this example. The last column is the deliverable: each type becomes test cases in the dataset and gets an evaluator whose rule names it.
Where it goes wrong
- Letting a judge do the first pass. A judge can only look for failures someone described to it. Error analysis is where the description comes from.
- Labeling the symptom. "Customer was angry" describes the outcome. Find the agent's turn that caused it, if there is one.
- Types too broad to check. If two people would disagree about whether a conversation belongs to a type, split or sharpen it.
- Reading once. New product features, new customers and new models produce new failures. Repeat on a fresh sample whenever something changes.
What interviewers are listening for
Asked how they would evaluate a new agent, strong candidates say they would read real conversations first and name the failures before choosing metrics. Even better is a specific example of a failure no standard metric would catch, and how reading found it.
Key takeaways
- Read and label before measuring; the failure list decides every later eval choice.
- Note the first point of failure, group notes into checkable types, rank by frequency and cost.
- Include passing conversations: a labeler who calls every unhappy customer a failure trains the loop on false alarms.