TL;DR: Model it as two stages, P(play) and E[watch time | play], train the watch-time head only on plays while correcting for the selection and position bias that created those plays, transform the heavy-tailed label (log1p or quantile buckets), and judge success on online A/B engagement with guardrails, not on offline MAE alone. The whole difficulty is that your label only exists for items users already chose.
How to approach it
State the objective the business actually wants before touching a model: not raw minutes (that rewards long, slow content and autoplay traps) but valued watch time, capped and possibly weighted by completion, satisfaction surveys, or downstream retention. Then ask the three questions that decide the design: what is the candidate set per request (tens of thousands of titles, not the whole catalog), what counts as a positive label and over what window (a 2-minute sample versus a finished episode), and what feedback do we log (impressions with position, plays, watch seconds, thumbs, abandons). Only then split the problem into play probability and watch-time-given-play, because those two have different biases.
A strong answer
Frame ranking score as score = P(play) * E[watch | play], an expected-watch-time estimate per candidate. Predicting watch time directly over all impressions is tempting and wrong: most impressions are not plays, so a single regressor spends its capacity learning the zero-inflated play decision and underfits the conditional watch-time it is supposed to estimate. Two heads, often a shared-bottom multi-task network, keeps each label clean.
The label is the hard part. Three biases distort it, and an honest answer names all three:
| Bias | Where it comes from | Correction |
|---|---|---|
| Selection bias | You only observe watch time for items the user chose to play; the rest are missing-not-at-random | Train the watch head only on plays; model P(play) separately; weight by inverse play propensity for the items you do see |
| Position bias | An item shown in slot 1 gets more plays and attention than the identical item in slot 30 | Inverse-propensity weighting by position, or a learned position-bias term concatenated at train and zeroed at serve |
| Survivorship | Long abandons look like short watches; a 90-minute film watched for 5 minutes is a failure, not a 5-minute success | Define the label as completion ratio or capped/valued seconds, not raw seconds; treat early abandon as a strong negative |
For the label transform, watch seconds are heavy-tailed: a few binge sessions dominate the mean and an L2 loss chases them. Predict log1p(seconds) or model completion ratio in [0,1], or bucket into quantiles and treat it as ordinal classification. Log regression with a calibrated exp-back-transform is the usual default; quantile buckets are more robust when the tail is extreme and you mostly care about ranking. Cap absurd values (autoplay-overnight 8-hour sessions) before they poison training.
Features and labels. User: long- and short-term watch embeddings, genre affinity, recency/time-of-day, device. Item: content embeddings, metadata, age since release, historical completion rate (a strong but leaky feature, use it carefully). Context: row/position, what surrounds it, session state. Critically, log the position of every impression at serve time; you cannot debias what you did not record. Cold-start items have no interaction history, so lean on content embeddings and metadata, and exploration (a small epsilon of random or Thompson-sampled placement) so they accumulate unbiased impressions instead of starving forever.
Offline metrics must match the two heads. For P(play): AUC and, more importantly, calibration (reliability curve, ECE), because the score multiplies into the final rank and a miscalibrated play probability silently reweights everything. For watch-given-play: MAE/RMSE on the transformed scale plus calibration of the back-transform. But the metric that actually predicts production is ranking quality, NDCG or a watch-time-weighted ranking metric on held-out sessions, ideally evaluated with counterfactual estimators (IPS or a doubly-robust estimator) so you are scoring the policy, not just refitting the logged behavior.
Online eval is the arbiter. Run an interleaving or A/B test and read valued watch time per user, plays per session, and next-day/next-week retention as the primary, with guardrails: total session length (so you do not just inflate one metric), abandon rate, catalog diversity/coverage, and a fairness check that new and niche titles still get impressions. Offline MAE going down while online engagement is flat or negative is the canonical outcome when you trained on a biased label and never corrected it; that gap is the whole point of the question.
Feedback loop and failure modes. The model picks what gets shown, those impressions become tomorrow's training data, and the loop self-reinforces: popular titles get more slots, get more plays, look even better. Counter it with exploration, propensity logging, and periodic audits of impression share by item age and popularity. Other failure modes: clickbait thumbnails that win P(play) but lose watch (why you optimize expected watch, not clicks), autoplay inflating seconds (cap and value the label), and stale embeddings after a content refresh.
What interviewers probe next
- "Offline MAE improved but the A/B lost engagement, what happened?" Most likely you optimized a biased label: improved fit to logged plays that themselves came from a biased policy. Check position/selection correction, check calibration, and trust the online watch-time and retention numbers over offline error.
- "Why not just predict clicks?" Clicks reward clickbait and autoplay; an item can win the play and lose the session. Expected watch time (or valued watch time) ties the objective to the outcome you actually want, which is why the watch head exists.
- "How do you handle a brand-new title with zero history?" Content embeddings plus metadata for the cold prediction, plus an exploration budget so it earns unbiased impressions; never let popularity priors starve it to death.
- "Your completion-rate feature is the strongest predictor, any concern?" It is partly a function of who the current policy already shows it to, so it leaks the policy's bias and can be stale; use it but monitor, and prefer features that are causal rather than self-fulfilling.
Common mistakes
Training one watch-time regressor over all impressions and letting the zero-inflated play decision swamp the conditional estimate. Ignoring that the label is missing-not-at-random and reporting MAE as if the watched set were representative. Never logging position, then claiming to debias it. Optimizing raw seconds so autoplay and 8-hour idle sessions dominate, instead of capping and valuing the label. Leaving the heavy tail untransformed so L2 chases binge outliers. And declaring victory on offline MAE without an online A/B, when the entire risk in this problem lives in the train/serve label gap.
Key takeaways
- Two stages: P(play) times E[watch | play]; never one regressor over all impressions.
- The label is missing-not-at-random; correct selection and position bias (IPW or learned position term) or your offline numbers lie.
- Transform the heavy-tailed label (log1p or quantile buckets), cap autoplay outliers, and value completion over raw seconds.
- Calibration feeds the rank; but the verdict is online valued watch time and retention with diversity and session-length guardrails.