TL;DR: ROC AUC measures ranking across all thresholds, and its false-positive-rate axis is normalized by the huge negative class, so it stays near 0.95 while the alert queue is 98% junk. Under heavy imbalance report PR AUC with its base-rate baseline, and present a confusion matrix at the one threshold you actually ship.
How to approach it
Define it operationally, not geometrically: ROC AUC is the probability that a randomly chosen positive scores higher than a randomly chosen negative. It measures ranking quality across all thresholds, which is exactly why it can be disconnected from what the customer experiences at the one threshold you actually deploy.
A strong answer
Explain the failure mode with numbers. Take 1,000,000 events with 1,000 positives (0.1% base rate, typical for fraud or security alerts). Pick the threshold that catches 95% recall (950 of the 1,000 frauds); at that operating point a 0.95 ROC AUC model can still sit at a 5% false positive rate, and 5% of 999,000 negatives is ~50,000 false alarms against those ~950 true hits. Precision ≈ 1.9%. The customer's analysts see ~50 junk alerts for every real one, and "0.95 AUC" sounds like a lie to them.
The cleanest way to expose the blindness: grow the negative class and watch what each number does (arithmetic computed, same 5% false positive rate and 950 catches throughout). At 999,000 negatives, 5% FPR is ~50,000 false alarms and precision is 1.9%. Make it 9,990,000 negatives, ten times more legitimate traffic, same model, same threshold: FPR is still exactly 5%, ROC AUC does not move at all, and precision collapses to 0.19%. The customer's queue got ten times worse while the headline metric stayed identical, because a rate over the negative class is invariant to the size of the negative class. That invariance is a feature when you want to compare models across datasets, and it is precisely the property that makes the metric deaf to what a review team experiences.
The reason: ROC's x-axis is false positive rate, normalized by the huge negative class, so massive absolute false-positive counts look tiny. The fix: under heavy imbalance, report PR AUC (average precision), both axes are computed relative to the positive class, so it reflects the alert-queue experience. In this example PR AUC might be 0.30 while ROC AUC is 0.95; the random-guessing baseline for PR AUC is the base rate (0.001), not 0.5, so always state the baseline alongside it.
Which metric to trust is a function of base rate and what the customer operates against:
Then bring it back to deployment: curves are for model comparison during development; customers live at a single operating point. So the artifact you actually present in a pilot readout is a confusion matrix at the chosen threshold plus the cost per error type, "at the threshold matching your 200-alerts/day review capacity, you catch 78% of fraud at 35% precision." That sentence lands; AUC doesn't.
What interviewers probe next
- "When is ROC AUC still the right choice?", balanced classes, or when you care about ranking across the full range (e.g., comparing model versions offline, scoring leads to sort a list).
- "Can two models have the same AUC but different value?", yes; one can dominate in the high-precision region the customer operates in. Look at the curve shape where you deploy, not the integral.
- "How do you pick the threshold?", from the customer's operational constraint (queue capacity, SLA) or cost asymmetry, then validate on a held-out time period.
- "The customer's exec wants one number for the steering deck", give recall at the operating precision, trended weekly. Pick the number that moves when the business outcome moves.
Common mistakes
Describing the ROC curve axes correctly but failing the "so what", interviewers at Google and Scale specifically listen for the imbalance caveat. Claiming AUC is threshold-free as if that's purely a virtue (it's also why it hides the deployed reality). Confusing precision with false positive rate. And reporting PR AUC without its base-rate baseline, which makes a 0.30 score look broken when it's actually 300× better than chance.