TL;DR: "Accurate" is a discovery prompt, not a spec. Run a four-step translation: name the decision the model drives, price each error, pick the metric that tracks the decision (often recall at an alert budget), and set a target against the current baseline, all agreed in writing before you build.
How to approach it
Recognize the genre: this is the FDE skill, translating vague business intent into a technical contract. Open with the move interviewers want to see: "I wouldn't choose a metric yet. I'd first ask: when this model is wrong, what happens, and what does each kind of wrong cost?"
A strong answer
Run a four-step translation, narrated with a concrete scenario, say an insurer wants to "accurately" flag claims for fraud review:
1. Identify the decision the model drives. Not "predict fraud" but "decide which claims a 12-person review team examines." The decision and its capacity constraint shape everything: 10,000 claims/week, team reviews 400.
2. Price the errors. Ask for numbers, even rough ones. Missed fraud: average $8,000 paid out. Wrongly flagged claim: ~$40 of reviewer time plus a delayed payout that drives complaints. The cost ratio (~200:1) tells you recall dominates, but the 400/week capacity caps how far you can chase it.
3. Pick the technical metric that tracks the decision. Here: recall at a fixed alert budget, "of all fraudulent claims, what % lands in your team's 400-claim queue?" That's measurable weekly, maps to dollars, and an exec can repeat it. Other translations from the same playbook: ranking problems → precision@k or NDCG; cost-asymmetric binary calls → recall at a precision floor (or expected-cost minimization directly); regression with asymmetric pain (under-forecasting stock-outs vs over-forecasting storage) → quantile loss, not RMSE.
4. Set the target with a baseline, and write it down. What does the current process catch? If manual selection catches 30% of fraud within the same 400 reviews and the model gets 55%, that's the pilot headline: "+83% fraud caught, zero added headcount." Agree on the number, the measurement window, and who computes it before building, this is the difference between a pilot that converts and one that ends in metric litigation.
One more multiplication converts the pilot headline into the number the CFO repeats, and it is worth doing in the readout. If roughly 1% of the 10,000 weekly claims are fraudulent, the weekly fraud pool is 100 claims, about $800,000 of exposure at the $8,000 average. Manual selection catching 30% recovers $240,000 of that; the model at 55% recovers $440,000, so the same 400-review budget now returns an extra $200,000 a week (arithmetic verified, and every input is one the customer gave you in step 2). Present it that way and the metric conversation is over: nobody at the steering meeting asks about F1 when the slide says the review team just got $10M a year more effective without hiring. That is also the honest test of whether your metric translation worked; if you cannot chain the technical metric back to dollars using only numbers the customer agreed to, the translation is not finished yet.
Close with the guardrail metric idea: pick one primary metric to optimize, plus guardrails you refuse to degrade (false-positive rate on VIP customers, latency, demographic parity if regulated). One target, several tripwires.
The translations from the same playbook, by output type:
| Decision/output type | Technical metric | Note |
|---|---|---|
| Cost-asymmetric binary call | Recall at a precision floor | Or expected-cost minimization directly; here, recall at a fixed alert budget |
| Ranking | Precision@k or NDCG | Scores the top of the list, not every row |
| Regression with asymmetric cost | Quantile loss | Not RMSE; under-forecast stock-outs vs over-forecast storage |
What interviewers probe next
- "The customer can't price the errors.", estimate together from proxies (average claim size, reviewer hourly cost), label them as assumptions, and revisit after two weeks of real data. A rough number beats no number.
- "Business metric and ML metric disagree after launch?", the business metric wins; instrument the funnel between them (model flags → reviews → confirmed fraud → dollars) to find where the chain breaks.
- "What if optimizing the metric causes gaming?", Goodhart's law; e.g., a deflection-rate target met by frustrating users into hanging up. Guardrails plus periodic qualitative review.
- "How does this change for a GenAI system?", same discipline, fuzzier labels: define a rubric, build a golden set, use human or LLM-judge evals, but still one primary number the exec tracks.
Common mistakes
Answering with a metrics glossary instead of a translation process. Accepting "accuracy" and optimizing it on imbalanced data. Choosing F1 to avoid the cost conversation, symmetric weighting is itself a cost assumption, just an unexamined one. And skipping the baseline: a model that's "87% accurate" is unsellable until you can say what the status quo scores.