ab testing

The staff-level version of 'evals first.' Four layers, each with a job, a cost, and a blind spot, plus judge calibration numbers and the data flywheel that ties production failures back into CI.

Standard A/B machinery assumes a crisp conversion event; agents give you fuzzy, delayed, gameable outcomes. The metric hierarchy, the reopen-window trick, and the sample-size reality check that win this question.

Most working data scientists state the definition wrong, and interviewers know exactly which wrong versions to listen for. The correct one-sentence definition, the VP translation, and the three classic traps.

"Run it two weeks and see" fails this question. Strong answers work the power calculation backwards from a minimum detectable effect, and know the duration rules that protect against lying-by-novelty.

Daily peeking can quietly triple your false-positive rate, and almost every customer does it. The math of why, the multiple-comparisons cousin, and how to deliver the bad news without losing the room.

The maturity exam of production ML: a model that's better on paper and worthless in production. Five distinct causes, the funnel diagnostic that separates them, and when the right call is don't ship.

Bandits sound strictly better and usually are not. The regret-vs-inference tradeoff, the three conditions that actually favor a bandit, and the production failure that makes adaptive allocation a debugging nightmare.

They answer different questions, and mixing them up costs you the round: one tests your engineering, the other tests your model's business impact. What each can and cannot catch, and the order a strong candidate always runs them in.

Anyone can train a pCTR model. The hard part is proving a change is good before it touches revenue, when the model sits inside an auction, the logs only show ads that won, and a 1% calibration error is real money. This is an eval question, not a model question.

New courses launch every week and most browsers are signed-out or brand new, so the interesting half of this problem is what you recommend when you have almost no behavioral signal. The other half is not optimizing engagement into a clickbait catalog.

The whole product lives or dies on a feedback loop measured in seconds: a swipe is a label, and the next video has to reflect it. The hard part is exploring enough to discover what someone likes without trapping them in a filter bubble or amplifying junk.

A search box is a recommender with a strong prior: the query. The hard parts are understanding ambiguous intent, retrieving in two stages under a tight latency budget, and learning from clicks without teaching the model that whatever you showed first is best.