position bias

A plausible win rate can hide a changed evaluation population. Reconstruct paired votes, preserve unresolved cases and decide what evidence could justify a model release.

Averaging two scalar scores tells you almost nothing. Blind pairwise battles plus a win-rate confidence interval do, and the result is only trustworthy once you've controlled position bias, verbosity bias, and rater fatigue.

A feed touches billions of candidate items and has tens of milliseconds to pick the next 10. The interview is about the two-stage architecture, what you optimize when engagement and integrity pull against each other, and the biases that quietly poison your training data.

Watch time is the label everyone optimizes and almost nobody measures cleanly. You only observe minutes for videos people chose to play, the distribution is brutally skewed, and the position they saw it in moved the number. The interview is whether you can predict a biased label honestly.

A search box is a recommender with a strong prior: the query. The hard parts are understanding ambiguous intent, retrieving in two stages under a tight latency budget, and learning from clicks without teaching the model that whatever you showed first is best.