TL;DR: They answer different questions and run in sequence, not as alternatives. Shadow mode mirrors live traffic to the new model with predictions logged but never acted on, so it tests engineering and skew at zero user risk but cannot prove the model is better. A/B testing splits live decisions to measure business impact with statistical rigor. Run shadow first to burn down engineering risk, then A/B (often via a small canary) to measure value.
How to approach it
A definitional screen with a judgment layer hidden inside. Anyone can define both; the rubric line is whether you know what question each answers and that they're sequential stages, not alternatives. Answer the definitions in two sentences each, then spend your time on what each one cannot catch.
A strong answer
Shadow deployment: the new model receives a copy of live traffic, its predictions are logged but never acted on, and the incumbent keeps making every real decision. Zero user risk by construction. What it tests is your engineering: does the model survive real payloads (nulls, weird encodings, traffic spikes), does it hit latency budgets in the actual serving container, do its online features match what it saw in training, and how does its score distribution compare to the champion's. Disagreement analysis is the high-value move, sample the cases where champion and shadow diverge and review them, because that's where the new model is either winning or broken.
A/B testing: live traffic is split, the new model makes real decisions for its slice, and you compare business metrics, conversion, fraud loss, watch time, with statistical rigor. This is the only way to measure impact, and the only way to see feedback effects: anything that depends on users or adversaries reacting to the model's decisions is invisible in shadow, where decisions never reach the world.
The asymmetry to state explicitly: shadow has no user feedback, so it cannot tell you the model is better, only that it's safe to try. A/B carries real risk, so it should never be your first production exposure. Hence the order that reads senior: shadow for one to two weeks to burn down engineering risk, then A/B (often via a small canary first) to measure value. Skipping shadow surrenders free skew detection; skipping A/B means you promoted on offline metrics and hope.
Two operational details that signal real experience: shadow mode doubles inference compute on shadowed traffic, so for expensive models you shadow a 10–20% sample rather than everything; and A/B splits should be randomized by entity (user, account) rather than by request, or one user gets inconsistent decisions and your measurement is contaminated.
The order a strong candidate runs them in:
Shadow answers "safe to try"; A/B answers "actually better." Skip either and you are guessing on one of those questions.
What interviewers probe next
- "What does shadow mode miss?", feedback loops: downstream teams acting on scores, fraudsters adapting, recommendation-driven behavior shifts. Decisions that never fire can't cause second-order effects.
- "When would you skip A/B testing?", when the decision isn't user-facing and offline plus shadow evidence suffices (an internal ranking refresh), or when labels arrive fast and a canary with guardrail metrics gives you the read cheaper. Say this as a trade-off, not a rule.
- "How long do you run the A/B test?", until a pre-computed sample size, not until significance appears; peeking at p-values daily and stopping on the first significant read is how teams ship noise. The inflation is not folklore, it simulates in a few lines and we ran it: 4,000 A/A experiments (both arms identical by construction, 5% conversion, 1,000 users per arm per day for 30 days), checking the p-value daily and stopping at the first p < 0.05. 26% of them found a "significant winner", five times the nominal 5% false-positive rate, in experiments where no difference existed to find. Thirty peeks means thirty chances for noise to cross the line, and noise takes them. That is the number to quote when a stakeholder asks why you will not call the test on day six's exciting dashboard, and the disciplined alternatives to offer are a fixed horizon or a sequential design (alpha-spending or always-valid p-values) that prices the peeking in. For low-base-rate outcomes like fraud, the honest fixed horizon can mean four to eight weeks.
- "Where does canary fit relative to these?", canary is a small-percentage live rollout with guardrails: operationally it sits between shadow (no live decisions) and a full A/B (powered measurement); it's about safety, not measurement precision.
Common mistakes
Presenting them as interchangeable options, the "which question does it answer" framing is precisely what's being screened. Claiming shadow mode validates accuracy gains (without labels and without behavior change, it can't). Forgetting the doubled compute cost. And describing request-level randomization for A/B tests, which any Netflix-style experimentation interviewer will flag immediately.