Core
A/B, Canary and Shadow Testing
Offline evals tell you a change is plausibly better; online testing tells you it actually is. A/B randomizes users between versions to measure real impact with statistical significance. Canary routes a small slice of live traffic to the new version to limit blast radius, and shadow mirrors real traffic to it with no user-facing effect so you can validate safely before anyone is exposed.
a free account unlocks the core curriculum tier · no card
Related concepts
Practise this in questions
MLOps & ML EngineeringWhat's the difference between shadow deployment and A/B testing a model? →easy · FreeMLOps & ML EngineeringWould you choose blue-green or canary deployment for a new model version, and why? →medium · Free with sign-inRAG & Agent System DesignA/B test a conversational agent when 'success' is fuzzy, resolution, CSAT, deflection without anger →hard · PremiumML System Design (Product)Design an evaluation framework for an ads-ranking system. →hard · FreeML System Design (Product)Design a recommendation engine for an online-course (or e-commerce) catalog. →hard · FreeML System Design (Product)Design the 'For You' ranking system for a short-video feed. →hard · Free with sign-in