AI Evals for FDEs: From Production Failures to Release Gates

Useful AI evaluations turn observed failures into repeatable engineering decisions. Review traces, define acceptance rules, build versioned cases and compare changes on the same work. Use code for exact checks and calibrated judgment for ambiguous language. Keep unsafe effects, missing evidence, cost and latency visible when deciding whether to release.

The asset is the accumulated record of what the application must do, how you checked it and which failures a change repaired. It remains useful when the model changes.

Consider Calder Equipment, the fictional support system in our Production AI Agents course. Its assistant can request an approved replacement. A polished reply saying “done” is insufficient: the request might have failed, succeeded twice or used expired approval. These outcomes require different tests.

Read failures before choosing the dashboard

Review a manageable batch of attempts, including successful cases and uncertain outcomes. A hundred traces can be a useful starting budget; it is not a statistical guarantee or a prerequisite for starting. Sample across customer segments and task types, and separately inspect consequential incidents. Record how you sampled before interpreting category frequencies.

First write concrete notes: “The service reply disappeared; the assistant issued a new request identity.” This is open coding: describe the observed problem before forcing it into a category. Then group notes by mechanism, an axial coding pass. “Unknown write treated as failure” is more actionable than “bad reasoning.” Allow multiple labels when one attempt contains several failures.

Count affected cases, not just annotations, and inspect impact alongside frequency. Twenty awkward greetings might cost less than one unauthorized replacement. The top recurring categories suggest quality measures; a rare forbidden effect can still become an independent release veto. Keep ambiguous cases for adjudication rather than inventing labels to finish the spreadsheet.

The thirty-failure prompt exercise develops this diagnostic habit. Fix the runtime or missing evidence when that is the cause; a prompt edit is only one possible intervention.

Give each dataset a job

A golden dataset needs inputs, acceptance predicates or rubrics, source provenance and useful slices. For an agent, also retain initial state, permitted tools, policy version and the independently observed outcome. “Expected output” can mean a state transition rather than one exact sentence.

Separate development cases, regression cases protecting known behavior, and fresh confirmation cases used to assess a candidate. Repeatedly inspecting a test set turns it into development feedback. Keep paraphrases and repeated attempts from the same source task in the same split; different text is not necessarily independent evidence.

The figure separates three reporting purposes. A traffic sample supports a workload estimate. A risk suite deliberately overrepresents dangerous cases. A diagnostic set concentrates on known failures. Combining them without explicit weights changes what the score means.

Version the cases, labels, rubric, judge, model revision, prompt, sampling settings, tool schemas and retrieval snapshot. Record retries and execution conditions. A hash identifies retained bytes; it cannot reconstruct an unpinned hosted service.

Synthetic cases help fill named gaps. Define dimensions such as intent × network condition × authorization state, select feasible combinations, then write natural inputs. Include both actions that should occur and actions that must be refused. Review generated labels independently and check for duplicates. A generator's breadth does not establish production frequency.

Choose a check that can observe the failure

Start with schema validation, required fields, identity checks and outcome assertions. For code or SQL, execute against controlled fixtures and inspect behavior. A query that runs can still compute the wrong business measure. A test using an external service, clock or shared database can still be flaky; “written in code” does not mean zero variance.

For RAG evaluation, label relevant evidence and score retrieval separately. Recall@k measures how much known relevant material appears in the first k results; MRR emphasizes the first relevant result; nDCG supports graded relevance and ranking quality. Incomplete relevance labels can make a good retrieval look bad. Freeze the corpus version used for the comparison.

Faithfulness asks whether answer claims are supported by the supplied context. It does not establish that the context is current, that the answer is complete or that it answers the user's question.

For agents, a reference trajectory is useful when the path is constrained. Otherwise, check necessary actions, forbidden actions and acceptable dependencies rather than demanding one exact tool sequence. Fourteen calls deserve investigation, but extra verification or recovery may be justified. LangChain's testing guidance distinguishes isolated unit checks, integration tests and trajectory evaluations; our agent evaluation concept connects these to deployment decisions.

Measure consistency across attempts

Run each case repeatedly with the serving configuration you intend to use. Retain temperature, sampling parameters, budgets and environment resets. A temperature of zero does not eliminate provider or infrastructure variation.

For independent attempts on one case with constant success probability p:

• At least one success in k attempts: 1 − (1 − p)^k.
• Success in every attempt: p^k.

At p = 0.8 and k = 5, these are approximately 99.97% and 32.77%. The τ-bench paper introduced pass^k to expose repeated-trial reliability. Neither formula should be applied blindly to an average across tasks with different difficulty or to correlated trials. For actual repeated observations, retain case identity and use the corresponding estimator, as in our pass@k and pass^k exercise.

Multi-turn simulations add hidden goals, corrections and interruptions that single prompts miss. Freeze the simulator's scenario and policy, inspect its conversations and compare its behavior with reviewed real interactions. Score the resulting state independently; the simulated customer declaring satisfaction is weak evidence. Use the simulated-user regression exercise to separate useful fault detection from unsupported prevalence claims.

Evaluate the judge before trusting its preference

Write a rubric a domain reviewer can apply. Build development and untouched confirmation examples with meaningful failures, split by source family. For a binary judge, define which label is positive. If positive means an acceptable response, report sensitivity on acceptable responses and specificity on unacceptable ones, plus counts and uncertainty. Accuracy alone can hide a judge that accepts nearly everything.

Require a concise evidence reference and verdict when that helps auditing. Test whether this improves agreement; a confident rationale is not a guarantee. Preserve “insufficient evidence” as a separate disposition. Pin the judge and rerun calibration when its inputs or rubric change. The calibration lesson includes a worked confusion matrix and an explicit missing-evidence boundary.

The MT-Bench judge study documents position and verbosity biases. Randomize answer order and compare both orderings. Map the second vote back to model identity. Keep consistent wins, ties, disagreements and invalid judgments in the report. Dropping disagreement can select the easy cases and inflate the apparent winner.

If longer answers win, examine whether extra detail is useful under the rubric. Compare length slices or apply a product length budget before generating both candidates. Truncating an already generated answer can remove a necessary qualification and change the object being judged. The pairwise judge debugging question works through the resulting denominator problem.

Separate a measured difference from a release decision

For 80 successes among 100 independent cases, a 95% Wilson interval is roughly 71.1%–86.7%, calculated using the NIST-described method. That interval describes one proportion. It does not directly test a change from 80 to 83 successes.

Compare matched outcomes. Three gains and no losses differ from thirteen gains and ten losses, even though both add three successes. A paired bootstrap resamples the same case indices for both versions; repeated trials and related examples require an appropriate task or family grouping. SciPy documents this paired resampling behavior. Set the meaningful effect size, uncertainty rule and number of planned comparisons before looking for a win.

Put exact safety checks and invalid-run detection in CI. Run relevant sliced evaluations when prompts, models, tools or retrieval change. For stochastic scores, use agreed tolerances and confirmation rules; rerunning until green is selection. Preserve missing outcomes instead of counting only completed attempts.

Operate the suite after release

Sample live traffic asynchronously, record selection probabilities where appropriate, and audit the judge's failures by category. A drift alert can reflect a changed workload, a changed judge or a changed system. Inspect those separately. Shadow candidates need isolated writes and bounded resource access; hiding their answers from users does not prevent side effects.

Report p50 and p95 latency, total recorded spend, unresolved attempts and cost per verified successful task. Include retries and failed attempts in the cost numerator. Keep the task mix and completion definition fixed when comparing versions. Lower cost with unchanged verified quality can be valuable; a cheaper system that silently hands more work to people has moved the cost elsewhere.

Turn incidents into minimal, sanitized, deduplicated regression cases after reviewing their labels and permissions. Keep fresh confirmation work separate. When a new model arrives, test a versioned candidate configuration against the baseline before considering a migration.

The free evaluation lesson shows how an aggregate winner can still fail a release gate. Continue with the Premium dataset project to construct the evidence: a coverage grid, source-family splits, executable manifest checks and a deliberately broken case you can explain.