TL;DR: Evaluate in three tiers. Offline, judge calibration and NCE/log-loss, not just AUC, because the predicted probability feeds the auction bid. Online, run A/B tests with explicit revenue-vs-user-experience guardrails plus long-run holdouts. And correct for the fact that logs only contain ads that won the auction, with counterfactual/IPS estimation, because naive offline replay is biased toward what the old model already showed.
How to approach it
The first move is to reframe: they did not ask you to build a pCTR model, they asked how you would know a change is good. So the answer is an evaluation stack, offline to online, that survives two facts that make ads different from a feed. First, the model's output is a probability that gets multiplied into a bid (bid = pCTR * advertiser_value in a roughly second-price auction), so being well-calibrated matters as much as ranking well, a systematic 5% over-prediction overcharges advertisers and distorts the whole marketplace. Second, your training and eval logs only contain ads that won the auction and got shown, so the data is selection-biased by the very system you are trying to evaluate. Cover the offline metrics, the auction interaction, the counterfactual problem, and the online experiment with guardrails.
A strong answer
Offline is the cheap fast gate, and the metric mix is the tell. AUC measures ranking (can the model order a click above a non-click) and it is necessary but insufficient, because two models with identical AUC can have wildly different calibration. So the headline offline metrics are calibration and NCE (normalized cross-entropy, the log-loss relative to a background-CTR baseline, which is the standard ads measure of how much the model beats just predicting the average). You check calibration both globally (predicted CTR vs actual over all impressions) and per-slice (by advertiser, placement, audience, device), because a model can be calibrated on average and badly miscalibrated for a segment that happens to be a major revenue source. Log-loss and PR-AUC round it out for the heavy class imbalance (clicks are rare).
| Tier | Metric | What it tells you | What it cannot tell you |
|---|---|---|---|
| Offline | AUC / PR-AUC | Ranking quality, ordering | Whether probabilities are correct |
| Offline | Calibration (ECE, reliability), NCE, log-loss | Are predicted probabilities true; do they beat baseline | The causal effect on revenue or users |
| Counterfactual | IPS / weighted replay | Estimated online effect from logged data | Anything outside the logged action space |
| Online | A/B: revenue per session, CTR, plus guardrails | The real, causal effect | Long-run / network effects (use holdouts) |
The auction interaction is what trips up people who have only done classification. The model does not directly choose what to show; its prediction enters a bid and the auction does. So a model that ranks better in isolation can still lose revenue if its calibration shifts where bids land, and you cannot evaluate the model in a vacuum, you have to evaluate the model-plus-auction system. This is also why offline replay is dangerous: if you re-score old logs with the new model, you only have labels for ads that actually won and were shown, you have no idea what would have happened for an ad the old model never surfaced.
That is the counterfactual problem, and the honest answer names it. Your logged data is a biased sample of the action space, so a naive "replay the logs and count clicks" overstates familiar choices. The fix is inverse-propensity scoring: if you logged the probability that each ad was served (the propensity), you can reweight by 1/propensity to build an unbiased estimate of how a new policy would have performed, optionally pairing IPS with a model-based correction (a doubly-robust estimator) to cut variance. The practical prerequisite is logging propensities and keeping a small randomized/exploration slice of traffic, because without any randomization the propensities for never-shown ads are zero and IPS cannot reach them. This counterfactual estimate is a better ship-to-A/B gate than offline metrics alone, but it is still an estimate, not the verdict.
Online is the verdict, and ads is where the guardrail tradeoff is most explicit because the headline metric is revenue and revenue can be bought by degrading the user experience. So you run an A/B on revenue per session (or per query), but you gate it on user-side guardrails: ad load, hide/report rate, organic engagement, and long-run retention. A change that lifts short-term revenue 2% while raising hide rate and depressing sessions is a loss, because ads revenue is a long game and an over-monetized surface bleeds users. For exactly the long-run and network effects that a two-week A/B cannot see (advertiser bidding adapts, users habituate), you hold back a small slice of traffic (often around 1%) from ads changes for months as a long-term holdout, and you measure the cumulative effect there. You also watch advertiser-side health (delivery, cost-per-result stability), since the marketplace has two sides and starving advertisers shows up as revenue loss later.
The close: the framework is layered on purpose. Offline calibration and NCE are a fast cheap filter, counterfactual/IPS estimates the online effect while respecting selection bias, and the A/B with revenue-vs-experience guardrails plus a long-term holdout is the only thing that earns a launch. Anyone who answers "we'd check AUC" has missed that the model lives inside an auction, the logs are biased, and the metric that matters is a causal one.
What interviewers probe next
- "AUC went up, do you ship?" No. AUC is ranking only; if calibration regressed, bids are wrong and the auction misprices, which can lose revenue even with better ordering. Check calibration/NCE, then the counterfactual estimate, then the A/B.
- "Your logs only contain ads that won the auction, what does that do to offline eval?" It selection-biases the data toward what the old model showed, so naive replay overstates the status quo. You log propensities, keep an exploration slice, and use IPS (with a variance-reducing model correction) to get an unbiased counterfactual.
- "Revenue is up 2% in the A/B, launch?" Only if the user guardrails (hide rate, ad load, sessions, retention) and advertiser health hold, and ideally the long-term holdout confirms it is not a habituation effect that decays. Short-term revenue bought with user experience reverses.
Common mistakes
Answering with AUC as the goal and ignoring calibration, when the prediction is a probability that gets multiplied into a bid. Evaluating the model as a standalone classifier and forgetting the auction it feeds. Replaying logs naively as if they were an unbiased sample, when they only contain auction winners. No propensity logging and no exploration, which makes any counterfactual estimate impossible. Treating revenue as the only online metric with no user-experience or advertiser guardrails. And declaring victory on a two-week A/B for a change whose real effects (bid adaptation, user habituation) only show up over months in a holdout.
Key takeaways
- Offline, calibration and NCE matter as much as AUC because the prediction feeds an auction bid; check calibration per slice, not just globally.
- The model lives inside the auction, so evaluate the model-plus-auction system, not a standalone classifier.
- Logs are selection-biased toward auction winners; log propensities, keep an exploration slice, and use IPS/doubly-robust for an unbiased counterfactual.
- Online A/B decides, gated on revenue-vs-user-experience guardrails plus a long-term holdout for effects a short test cannot see.