Measurement validity
The judge agrees 94% of the time
Four of ten real failures still slip through. Read the confusion matrix before putting the judge in a release gate.
← The FDE reading roomEvals
The evaluation field guide / November 2026
Evals
for FDEs
Build evaluations you can trust, from first traces to production.
Find the failures that matter. Build useful datasets, validate your graders and test what an agent actually does. Follow one customer workflow through measurement, release decisions and production monitoring.
Every answer, concept and course, every paper breakdown, all published video lessons, hands-on FDE Lab missions, Premium PDF guides and companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.
Study alongside free video lessons.
Premium guide PDFs and companion files require active paid Premium or approved complimentary access. Referral Premium includes the rest of the site, but excludes these downloads.
74 pages · 32 chapters · PDF + code + worksheets
Design / measure / release / observe
- pages in a six-part journey
- 74
- original vector diagrams
- 15
- tests in the local companion
- 44
- primary references
- 40
What is an AI evaluation?
A repeatable procedure for judging a defined AI system under stated conditions, using specified inputs, evidence, grading and a decision rule.
A good evaluation makes the decision inspectable. This guide shows how to connect a customer requirement to a case, an observation, a grade and a justified action. It also explains what a score cannot establish.
Three complete pages, free to read
Inspect the method and the evidence.
Inspect a judge confusion matrix, compare two definitions of reliability and read a release-gate design. Each preview opens an actual PDF page at full size.
Page 26 / Sample
When 94% agreement misses the point
Read the confusion matrix and separate overall agreement from failure detection.
Open the page at full size ↗
Page 30 / Sample
One success is different from consistent success
Compare pass@k and pass^k with explicit assumptions and denominators.
Open the page at full size ↗
Page 44 / Sample
Decide what must block a release
Keep hard constraints, invalid trials and paired quality evidence visible.
Open the page at full size ↗
32 chapters / six parts
One learning journey, from customer need to release evidence.
Define the evidence
Start with a customer decision, acceptance criteria and real failures. Build versioned datasets with explicit coverage and permissions.
- 01
The decision before the score
p. 6 - 02
Write the acceptance contract
p. 8 - 03
Read failures before naming metrics
p. 10 - 04
Capture evidence you can trust
p. 12 - 05
Build a dataset with a purpose
p. 14 - 06
Generate and label useful cases
p. 16
Measure with care
Separate retrieval and generation, validate an LLM judge, compare paired results and understand repeated-run reliability.
- 07
Start with executable evidence
p. 18 - 08
Evaluate retrieval and answering
p. 20 - 09
Choose metrics and denominators
p. 22 - 10
Design a judge with a narrow job
p. 24 - 11
Validate the measurement instrument
p. 26 - 12
Compare systems with uncertainty
p. 28 - 13
Measure repeated-run reliability
p. 30
Evaluate acting systems
Check persistent effects, trajectories, simulations, memory, multi-agent handoffs, coding, browser, voice and visual tasks.
- 14
Grade agent outcomes and paths
p. 32 - 15
Build simulators and resettable worlds
p. 34 - 16
Test memory and multi-agent handoffs
p. 36 - 17
Evaluate coding and browser work
p. 38 - 18
Evaluate voice, vision and documents
p. 40
Release and observe
Build the harness, gate changes, sample production and measure cost per successful task. Improve without training on the exam.
- 19
Build the evaluation harness
p. 42 - 20
Gate changes without gaming the test
p. 44 - 21
Watch production with a sampling plan
p. 46 - 22
Measure cost, latency and human work
p. 48 - 23
Select models and read benchmarks
p. 50 - 24
Improve without training on the exam
p. 52
Protect the evaluation
Threat-model the evaluator, test action boundaries and retain evidence for governance and domain-specific review.
- 25
Red-team actions and the evaluator
p. 54 - 26
Governance and domain-specific evidence
p. 56
Choose and implement
Compare current cloud and open tools, run the local companion, deliver an evaluation programme and work through ten decision clinics.
- 27
Cloud tooling in November 2026
p. 58 - 28
Choose open tools and own the assets
p. 60 - 29
Run and break the companion
p. 62 - 30
Deliver an evaluation programme
p. 65 - 31
Practice the difficult decisions
p. 67 - 32
Worksheets and your next project
p. 69
Defend the decision when the easy answer fails.
One fictional support-and-refund workflow carries the method through the book. Ten decision clinics ask you to defend a measurement, investigate an exception and explain the remaining uncertainty.
Agent effects
The refund reply disappears
A retry looks sensible but creates a second refund. Inspect operation identity, attempted authority and the final ledger.
Evidence accounting
The better candidate cannot ship
The aggregate score improves, but a fixture remains invalid. Preserve the missing evidence and explain the blocked decision.
Included / executable companion
Change a condition. Explain the result.
Run two scripted candidates against a resettable refund service. Lose a reply, inspect a duplicate effect and distinguish a blocked unsafe attempt from compliant behavior. Calculate retrieval metrics, repeated-run reliability and a paired comparison. The archive passed 44 tests and 8 deliberate mutation checks.
Python and its standard-library SQLite support are enough for the local track. The walkthrough explains commands, expected results and limitations. The demonstration deliberately remains blocked on an invalid fixture so you can inspect how missing evidence affects a release.
8 editable operating worksheets
- Acceptance contract and dataset card
- Error-analysis worksheet and judge card
- Paired experiment and release memo
- Production sampling and monitoring plan
- Threat review and governance evidence
- Tool selection and exit plan
Teaching code and synthetic data. No hosted execution, production credentials or model account required for the local track.
Continue from a chapter into practice.
These paths connect the guide to focused concepts, questions and course labs. The same links are clickable inside the PDF.
Chapters 1–6
Build the evidence foundation
Chapters 7–13
Measure and validate
Chapters 14–18
Test acting systems
Chapters 19–26
Release and monitor
Chapters 27–32
Run and adapt the kit
A shared vocabulary for the evidence
- AI evaluation
- A repeatable procedure for judging a defined AI system under stated conditions, using specified inputs, evidence, grading and a decision rule.
- LLM-as-a-judge
- A model used to apply a written evaluation criterion to supplied evidence. Its error rates and decision coverage must be validated against reviewed labels.
- Agent evaluation
- An assessment of an agent's outcomes, action history and operating constraints, including authoritative state and attempted behavior.
- pass@k and pass^k
- At-least-one success and all-run success across k attempts, respectively. They answer different reliability questions and require explicit sampling assumptions.
- Evaluation release gate
- A predeclared decision rule that combines evidence validity, hard constraints, quality comparisons and operating limits before a change reaches users.
Questions about the guide
What are evals in AI engineering?
A repeatable procedure for judging a defined AI system under stated conditions, using specified inputs, evidence, grading and a decision rule. They can diagnose failures, detect regressions, compare capabilities or support a release decision. Those uses need different case mixes and evidence.
Does this cover LLM, RAG and agent evaluations?
Yes. The guide separates language-model outputs, retrieval evidence, grounded generation and acting systems. It covers state and trajectory checks, memory, simulated users, multi-agent handoffs, coding, browser, voice and visual tasks.
Do I need a statistics background?
Basic Python and familiarity with requests and tool calls are enough to begin. Worked examples explain denominators, confidence intervals, paired comparisons, pass@k and pass^k. The companion calculates the examples so you can inspect each step.
What can I run locally?
The Python/SQLite companion uses scripted candidates and synthetic data, without API keys or extra packages. Its extracted archive passed 44 tests and detected 8 deliberate mutations. It demonstrates refund-state verification, idempotent retries, attempted-authority checks, retrieval metrics, judge calibration and a paired release decision.
Does the companion evaluate a live model?
No. Its candidates are controlled scripts so the measurement behavior is reproducible. It does not demonstrate production authentication, a live judge, a durable handoff queue or cloud deployment. The guide explains the additional adapters and evidence needed for those uses.
How does the guide handle unreliable LLM judges?
It defines narrow rubrics, protects evidence boundaries, measures pass and failure recall separately, retains unknown outcomes and checks position bias. Overall agreement is never treated as proof that a judge catches costly failures.
Which evaluation tools are covered?
The November 2026 tooling chapters compare official Google, AWS, Microsoft and OpenAI interfaces and lifecycle notices, then open tools including Inspect, Ragas, DeepEval, promptfoo, Phoenix, Langfuse, MLflow, TruLens, Braintrust and Opik. Selection starts with your evidence and data boundary, and dated claims link to primary documentation.
What is included in the guide download?
The 74-page PDF, a separate tested companion ZIP, 8 editable worksheets, 15 original diagrams and ten decision clinics. The running support-and-refund scenario and all worked results are synthetic teaching material.
Is the complete guide included in my plan?
Premium guide PDFs and companion files require active paid Premium or approved complimentary access. Referral Premium includes the rest of the site, but excludes these downloads. Three complete sample pages are public. Downloads follow the site's guide entitlement and any applicable pass limits. Referral-only online Premium does not include guide downloads.
How does it connect to the website?
Chapter links lead to the eval loop, judge calibration, Production AI Agents, Harness Engineering, Hard Deployments, interview questions and interactive FDE Lab missions. Each online resource keeps its displayed access level.
How was the guide produced?
AI tools assisted research, writing, editing and code. The cover uses locally typeset text and original SVG artwork; the diagrams are authored vector figures. Automated tests and assistant review do not imply independent human editorial approval. The publication notice explains provenance, permitted use and limitations.