TL;DR: Predict pCTR and pCVR as two calibrated models whose product feeds the auction's expected-value bid, so calibration (not just ranking) is the primary requirement. Handle delayed conversions with a model that treats unconverted-so-far clicks as censored rather than negative, solve cold-start for new ads/advertisers with content and hierarchical priors plus exploration, and gate launches on online revenue and ROI with a spend guardrail.
How to approach it
Anchor on what the prediction is for: in a per-impression auction, the bid for an ad is roughly pCTR * pCVR * advertiser_value, so these probabilities are priced, not just ranked. That single fact reorders the priorities: calibration matters as much as discrimination, and a model that ranks perfectly but is 2x off in absolute probability systematically mis-bids. Clarify the latency budget (scoring hundreds to thousands of candidate ads per request in low tens of milliseconds), the conversion definition and its attribution window (a purchase within 7 days of click is a very different label from an add-to-cart in-session), and how fresh ads and advertisers enter the system. Then design two stages, CTR and CVR, with calibration and delayed feedback as first-class concerns.
A strong answer
Model the funnel in two stages because the conditioning and the feedback timing differ. pCTR = P(click | impression), trained on impression-level logs, label available within seconds. pCVR = P(conversion | click), trained only on clicks (clicks are the impressions for this head), label arriving anywhere from minutes to days later. Multiplying gives expected conversions per impression, which the auction turns into a bid. Keeping them separate lets each head use its own features and its own debiasing, and avoids drowning the rare conversion signal in the much larger click decision.
Calibration is the headline. Because the bid is a function of the probability, you optimize for low absolute error, not just ranking. Train with logistic loss, then add an explicit calibration layer (Platt scaling or isotonic regression) and monitor expected calibration error and reliability curves per major segment (device, vertical, country), since global calibration can hide segment-level skew that mis-prices a whole advertiser class. Also calibrate per slot/position because the same ad has different CTR by placement; position bias here is both a debiasing problem (use a position-bias model or IPW) and a calibration problem. Report AUC and log-loss, but treat calibration as the launch-blocking metric.
Delayed feedback is the CVR head's defining challenge. You cannot wait the full attribution window to train, traffic and creatives move too fast, but if you label every not-yet-converted click as negative, you systematically under-predict CVR (many of those clicks will convert tomorrow). The label is censored, not negative. The standard handling is a delayed-feedback model: jointly model the conversion probability and the delay distribution (for example an exponential time-to-conversion) so a recent click with no conversion yet is treated as "possibly converting later" rather than a confirmed non-converter, and importance-weight or correct the loss accordingly. Operationally, retrain frequently on a sliding window and re-attribute conversions as they arrive. Pick the attribution window deliberately: too short censors real conversions, too long delays learning and complicates measurement.
Features and crosses. User: history, interests, recent intent signals. Ad/advertiser: creative embeddings, category, historical CTR/CVR with smoothing. Context: query/page, device, placement, hour. The single highest-leverage piece is feature crosses: user-x-ad-category, query-x-creative, geo-x-vertical. Classic systems learned these with massive sparse logistic regression or factorization machines; modern ones use embeddings plus a deep cross network (DCN/DeepFM style) so crosses are learned rather than hand-enumerated. Hash high-cardinality IDs and watch for the cross explosion blowing up the feature space.
Cold start is acute because pricing a new ad wrong wastes real money. A brand-new ad or advertiser has zero clicks and conversions, yet the auction needs a number on impression one. Use:
| Cold-start case | What you serve | Mechanism |
|---|---|---|
| New creative, known advertiser | Borrow from advertiser/category priors | Hierarchical/empirical-Bayes smoothing up the ad to campaign to advertiser to category tree |
| Brand-new advertiser | Content-based estimate | Creative + landing-page embeddings, similar-advertiser lookup |
| Any new entity | Earn its own data | Explicit exploration budget (epsilon or Thompson sampling) so it accrues unbiased impressions instead of being starved |
The exploration point is the senior tell: a pure exploit policy never gives a new ad the impressions it needs to prove out, so it dies regardless of quality. Budget exploration and bound the spend so a bad new ad cannot burn the budget while learning.
Online eval and feedback loops. Offline, optimize log-loss/calibration/AUC on a held-out window. Online, A/B on the metrics that pay: revenue per mille, advertiser ROI/ROAS, total conversions, with guardrails on user experience (ad load, relevance, hide/complaint rate), advertiser delivery fairness, and budget pacing. The feedback loop is real and self-reinforcing: the model decides which ads are shown, those become tomorrow's training data, so an under-served good ad never gets a chance to correct its own under-prediction. Exploration plus propensity logging keep the loop honest. Watch for the failure where optimizing short-term clicks degrades long-term advertiser ROI and they churn, which is why CVR and downstream value, not CTR alone, drive the bid.
What interviewers probe next
- "A fresh advertiser with no history, what pCVR do you serve on impression one?" A content-and-prior estimate (creative/landing-page embeddings, similar-advertiser and category priors via hierarchical smoothing), plus an exploration budget with a spend cap so it earns its own calibrated data without burning money.
- "Why is calibration more important here than in a typical classifier?" The probability is a price input to the auction bid, not just a sort key; a 2x calibration error mis-bids every impression, so you add isotonic/Platt calibration and monitor ECE per segment.
- "How do you train CVR when conversions arrive days later?" Treat unconverted clicks as censored, not negative; use a delayed-feedback model that jointly estimates conversion and delay, retrain on a sliding window, and re-attribute conversions as they land.
- "Your AUC went up but revenue dropped, why?" AUC measures ranking, not pricing; calibration likely drifted (often per-segment), so the auction over- or under-bid even though ordering improved. Check reliability curves by segment.
Common mistakes
Optimizing AUC and never mentioning calibration, when the output is a bid input. Labeling not-yet-converted clicks as negatives, which biases pCVR downward (delayed feedback ignored). Collapsing CTR and CVR into one model so the rare conversion signal gets swamped. No cold-start plan, so new ads/advertisers are mis-priced or starved with no exploration. Ignoring position bias, which corrupts both ranking and calibration. And evaluating only offline, when the auction interaction and the feedback loop mean only an online A/B on revenue and advertiser ROI tells you the truth.
Key takeaways
- The prediction is a price (bid ~ pCTR * pCVR * value), so calibration, monitored per segment with ECE, is as critical as AUC.
- Two stages: pCTR on impressions, pCVR on clicks; do not merge them.
- Conversions are delayed; treat unconverted clicks as censored with a delayed-feedback model, not as negatives.
- Cold-start new ads/advertisers with content and hierarchical priors plus a spend-capped exploration budget; judge launches online on revenue and ROI with UX and pacing guardrails.