Offline vs Online Evaluation
Offline evaluation scores a change against a fixed golden set: fast, cheap, repeatable, and runnable in CI before anything ships. Online evaluation measures the change on real traffic and real users, usually via A/B, and is the only true read on impact. The two are not interchangeable: offline gains routinely fail to hold online because of distribution shift and metric gaming. The discipline FDE loops test is using offline to gate and online to confirm.
Every answer, concept and course, every paper breakdown, all published video lessons, hands-on FDE Lab missions, Premium PDF guides and companion files, the full practice-test bank and work-sample downloads. Referral Premium excludes guide PDFs and their companion files.
6 months · One payment · No auto-renewal
Study alongside free video lessons.
Related course lessons
- Read an agent evaluation without losing the failures →
22 min · Free
Production AI Agents for FDEs · syllabus - An evaluation becomes a harness the day it can block a release →
24 min · Premium
The FDE Engagement · syllabus - Measure the baseline and distinguish time, capacity and cash →
28 min · Premium
The FDE Engagement · syllabus