19Design an A/B test for a customer's new recommendation model. How long do you run it, and on how many users?▼medium★ EssentialMetaGoogleDatabricks1 replies○ sign in"Run it two weeks and see" fails this question. Strong answers work the power calculation backwards from a minimum detectable effect, and know the duration rules that protect against lying-by-novelty.Open full answer →
21The customer can't randomize, every user must get the new model. How do you measure whether it worked?▼hardGoogleDatabricksMicrosoft1 replies◆ premiumEnterprise reality: legal, fairness or ops constraints kill the A/B test, but the exec still demands proof of impact. The quasi-experimental toolkit, and the honest caveats, that let you answer anyway.Open full answer →
37The customer's marketplace has network effects, so a user-level A/B test is biased. How do you measure the model's impact?▼hardMetaGoogleDatabricks1 replies◆ premiumWhen treatment leaks between units, a clean A/B lies in both directions. The interference taxonomy, when to reach for switchback vs cluster vs geo designs, and the analysis trap that makes naive standard errors useless.Open full answer →
39The customer wants to test five model variants without losing money on the bad ones. A/B test or a bandit?▼hardMetaGoogleDatabricks1 replies◆ premiumBandits sound strictly better and usually are not. The regret-vs-inference tradeoff, the three conditions that actually favor a bandit, and the production failure that makes adaptive allocation a debugging nightmare.Open full answer →
