TL;DR: Split it into cheap candidate generation (two-tower retrieval over an ANN index, a few thousand items) and an expensive ranker that scores a few hundred. Train the ranker on implicit feedback to predict multiple actions (click, dwell, like, follow, plus integrity signals), combine them with a weighted objective, correct position bias in the labels, and decide everything with online A/B tests gated on engagement and integrity guardrails.
How to approach it
Name the constraint first: you cannot score a billion-item corpus per request in 30 to 50 ms, so the architecture is two stages before it is any particular model. Ask what "good" means for this product (raw time-spent, meaningful interactions, creator-side health) because that decides the objective, not the loss function. Then ask what feedback you actually log, because implicit signals are what you train on and they are biased by what the old model already showed. Cover retrieval, label and feature design, the multi-objective ranker, and how you measure all of it online.
A strong answer
Stage one is candidate generation. From a corpus of hundreds of millions of items you narrow to a few thousand in single-digit milliseconds, usually by blending sources: a two-tower model (a user tower and an item tower trained so their dot product approximates engagement, served through an ANN index like HNSW or ScaNN), plus follow-graph items, plus fresh and trending pools. Two-tower is the workhorse because the item tower is precomputed offline and the user vector is one forward pass at request time, so retrieval is an approximate-nearest-neighbor lookup, not a model scoring billions of rows.
Stage two is the ranker, and it earns the compute because it only sees a few hundred candidates. Here you use a richer model (a gradient-boosted tree on a smaller surface, but at this scale a multi-task DNN) with full cross features: user history, item content embeddings, author affinity, recency, and crucially the user-by-item interactions the two towers cannot represent because they never mix until the dot product.
The hard part is labels. You have no explicit ratings, only implicit feedback, and it is noisy in specific ways. A click is weak (clickbait gets clicks), so you predict several heads: p(click), p(long-dwell), p(like), p(follow author), p(hide/report). Dwell needs a threshold or it rewards slow-loading confusion. Negatives are a trap: a non-click is not a true negative, the user may never have seen the item below the fold, so you treat displayed-but-not-engaged as a soft negative and sample un-displayed items as easy negatives for retrieval.
Then multi-objective, because optimizing one head wrecks the product. Raw p(click) breeds clickbait; raw dwell breeds doomscrolling that tanks next-day retention. So the serving score is a weighted combination, roughly score = w1*p(click) + w2*p(dwell) + w3*p(like) + w4*p(follow) - w5*p(report), with the integrity term subtracted. The weights are not learned end-to-end against one metric; they are tuned through online experiments against a basket of metrics including retention, because that is the only thing that tells you whether you traded a healthy feed for a click bump.
Now position bias, the failure that quietly corrupts everything. Items shown at the top get more clicks regardless of relevance, so if you train naively the model learns "top position is good" and reinforces whatever it already ranked highly. Two standard fixes: include position as a feature at training time and zero it out (or set it to a fixed value) at serving, so the model attributes the click to relevance not slot; or weight examples by inverse propensity, dividing each label by the estimated probability that the item was examined at that position. The same logic covers selection bias: your logs only contain items the previous model chose to show, so you periodically inject a small randomized or exploration slice to collect unbiased data and keep the feedback loop from collapsing into a monoculture.
Offline you tune with ranking metrics; online you decide with experiments.
| Layer | What you measure | Metric | Decision it drives |
|---|---|---|---|
| Retrieval | Did the good item even reach the ranker | recall@k, coverage | Candidate-source mix |
| Ranker (offline) | Ordering and probability quality | NDCG, AUC, calibration (ECE), log-loss | Ship-to-A/B gate |
| Online (causal) | Real user and creator effect | A/B on engagement, next-day retention, report rate | Launch or kill |
Calibration matters more than people expect because the scores feed a weighted blend and downstream ads/notification logic: if p(click) is not a true probability, the weights mean nothing across heads. So you check expected calibration error, not just AUC.
The close: nothing ships on an offline number. A model with higher NDCG can lose the A/B because it pushes engagement bait that lifts CTR and drops 28-day retention. So you run an A/B with explicit guardrails, integrity report rate, session length, retention, creator reach, and you treat a guardrail regression as a launch blocker even when the headline metric is up.
What interviewers probe next
- "CTR went up but sessions got shorter, do you ship?" No. CTR is a proxy; you optimized clickbait. Look at the retention and dwell guardrails in the same A/B, and if they regressed, the headline win is a loss. This is the question they are really asking.
- "How do you cold-start a new user or a brand-new item?" New user: lean on context and popularity priors, explore aggressively, personalize as signal accrues over the first sessions. New item: content embeddings (image/text/audio) so the item tower places it without interaction history, plus a small exploration budget so it can earn impressions.
- "Your training data is generated by your own model, how do you not collapse?" Acknowledge the feedback loop, then inject randomized/exploration impressions, use inverse-propensity weighting on logged data, and monitor diversity and catalog coverage as drift alarms.
Common mistakes
Proposing a single giant model that scores the whole corpus, which is infeasible at feed latency and shows you skipped the two-stage insight. Optimizing one engagement head (usually click) and ignoring that it degrades the product. Treating every non-click as a hard negative, which poisons training with items the user never saw. Forgetting position bias entirely, so the model just learns to trust its own past rankings. Quoting NDCG or AUC as if it were a launch decision, when feed launches are won and lost in the A/B on retention and integrity guardrails. And no exploration, so the feedback loop narrows the catalog into a self-reinforcing monoculture.
Key takeaways
- Two stages: two-tower retrieval over an ANN index, then a heavy multi-task ranker on a few hundred candidates.
- Implicit labels are biased; correct position bias (position as a train-time feature, zeroed at serving, or IPS) and selection bias (exploration slice).
- Optimize a weighted multi-objective with an integrity term subtracted, not raw click; tune the weights online.
- Decide with A/B tests gated on retention and integrity guardrails, not offline NDCG; keep calibration honest because heads get blended.