ML System Design (Product)

A feed touches billions of candidate items and has tens of milliseconds to pick the next 10. The interview is about the two-stage architecture, what you optimize when engagement and integrity pull against each other, and the biases that quietly poison your training data.

A track lasts three minutes, a session lasts an hour, and a new song has zero plays the day it drops. The interview is about blending collaborative filtering with audio content embeddings, surviving cold start on both sides, and reading a skip as the loud negative it is.

Anyone can train a pCTR model. The hard part is proving a change is good before it touches revenue, when the model sits inside an auction, the logs only show ads that won, and a 1% calibration error is real money. This is an eval question, not a model question.

Watch time is the label everyone optimizes and almost nobody measures cleanly. You only observe minutes for videos people chose to play, the distribution is brutally skewed, and the position they saw it in moved the number. The interview is whether you can predict a biased label honestly.

The positive class is maybe 1 in 1,000, your labels arrive late and noisy, and the moment you ship a model the adversary probes it. This is the rare ML problem where the data is actively trying to beat you, so the design is as much about labels, latency, and enforcement cost as about the classifier.

This model's output is not a ranking, it is a price input. A miscalibrated CTR makes the auction bid wrong, so you overpay or underdeliver. Add conversions that land days after the click and advertisers with zero history, and calibration plus delayed feedback become the whole interview.

An ETA is a promise. The interview is whether you model it as a point estimate (and apologize when wrong) or as a distribution where p90 lets you under-promise, plus how you fold live traffic and finished trips back into the model.

There are millions of landmarks, most with a handful of photos, and the next photo might be of something not in your catalog at all. A flat classifier dies on the long tail and never says 'I don't know.' The interview is embeddings plus retrieval plus a confident refusal.

Fraud is well under 1% of transactions, the labels arrive weeks late as chargebacks, and the fraudsters adapt to your model on purpose. Optimizing accuracy gives you a model that approves everything. The interview is dollars-at-risk thresholding, label delay, and a review queue with a fixed headcount.

New courses launch every week and most browsers are signed-out or brand new, so the interesting half of this problem is what you recommend when you have almost no behavioral signal. The other half is not optimizing engagement into a clickbait catalog.

The whole product lives or dies on a feedback loop measured in seconds: a swipe is a label, and the next video has to reflect it. The hard part is exploring enough to discover what someone likes without trapping them in a filter bubble or amplifying junk.

A search box is a recommender with a strong prior: the query. The hard parts are understanding ambiguous intent, retrieving in two stages under a tight latency budget, and learning from clicks without teaching the model that whatever you showed first is best.