"Run it two weeks and see" fails this question. Strong answers work the power calculation backwards from a minimum detectable effect, and know the duration rules that protect against lying-by-novelty.
more free answers with an account · no card
← Machine Learning & Data Science / 19
medium★ EssentialMetaGoogleDatabricks
Design an A/B test for a customer's new recommendation model. How long do you run it, and on how many users?
"Run it two weeks and see" fails this question. Strong answers work the power calculation backwards from a minimum detectable effect, and know the duration rules that protect against lying-by-novelty.
Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.
UP NEXT ON YOUR JOURNEY
Next in this trackThe customer checked the A/B dashboard daily and stopped the test the day it hit significance. What's wrong, and what do you tell them?Next in this trackThe customer can't randomize, every user must get the new model. How do you measure whether it worked?Next in this trackSix months after deployment, the CFO asks: 'What has this model actually earned us?' How do you answer?
DISCUSSION · 0
No comments yet — be the first to share your approach.