"Run it two weeks and see" fails this question. Strong answers work the power calculation backwards from a minimum detectable effect, and know the duration rules that protect against lying-by-novelty.
more free answers with an account · no card
medium★ EssentialMetaGoogleDatabricks
Design an A/B test for a customer's new recommendation model. How long do you run it, and on how many users?
"Run it two weeks and see" fails this question. Strong answers work the power calculation backwards from a minimum detectable effect, and know the duration rules that protect against lying-by-novelty.
Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.
Related course lessons
Keep practising
Next in this trackThe customer checked the A/B dashboard daily and stopped the test the day it hit significance. What's wrong, and what do you tell them?Free with sign-inNext in this trackThe customer can't randomize, every user must get the new model. How do you measure whether it worked?PremiumNext in this trackSix months after deployment, the CFO asks: 'What has this model actually earned us?' How do you answer?Premium
DISCUSSION · 0
No comments yet — be the first to share your approach.