TL;DR: Do not optimize accuracy or even a fixed probability cutoff; threshold on expected dollar loss, blocking a transaction when (fraud probability times amount) exceeds the cost of a false decline, and route the gray zone to a human review queue sized to actual headcount. The hard parts are extreme imbalance, labels that arrive weeks late as chargebacks, and an adversary who adapts to whatever you ship.
How to approach it
Lead with why accuracy is a trap: fraud is well under 1% of transactions, so a model that approves everything scores over 99% and catches nothing. The objective is money, not classification rate, so frame it as minimizing expected loss: false negatives cost the transaction amount plus chargeback fees; false positives cost a declined good customer (lost revenue now, sometimes churn forever). Clarify the operating reality: there is a real-time decision in the payment path with a tight latency budget, an async human review queue with finite headcount, and labels that lag by weeks. Those three constraints, plus the adversary, shape every later choice.
A strong answer
Architecture is two-tier. A real-time scorer sits in the authorization path and must return in roughly 50 to 100ms, so it is a fast model (gradient-boosted trees or a compact net) over precomputed features. Its output is not a single accept/decline; it splits into three zones by expected dollar loss: clearly safe to approve, clearly block, and a gray middle routed to asynchronous review (step-up auth like 3-D Secure, a hold, or a human analyst). The threshold between zones is set by economics: block when p(fraud) * amount > cost_of_false_decline, which means a $5,000 transaction gets blocked at a far lower fraud probability than a $5 one. The same probability is not the same decision; the dollar amount is part of the threshold.
Features are where fraud models win or lose, and the strongest ones are velocity and graph features, not the raw transaction fields. Velocity: count of transactions from this card, device, or IP in the last minute/hour/day; amount spent in the last hour versus the account's baseline; number of distinct cards on this device. Graph features: is this device, email, or shipping address linked to other accounts that charged back? Fraud rings share infrastructure, so a transaction that looks clean alone is damning when the device has touched twelve charged-back accounts. Add behavioral signals (time since signup, typing/session anomalies), device fingerprint, and BIN/geo mismatches. These features must be computed consistently online and offline (a feature store) or you get train/serve skew that silently tanks production.
The label is the subtle, defining problem. Ground truth is mostly the chargeback, which arrives 2 to 12 weeks after the transaction. So at any moment your most recent weeks of data are unlabeled or partially labeled: a recent fraud spike is invisible in your training set until the chargebacks land. Consequences: a model retrained today learns from a fraud distribution that is one to two months stale, and naive metrics on recent data look great because the fraud has not been labeled yet. Mitigations: treat label maturity explicitly (only call a transaction "good" after the chargeback window closes), use analyst reviews and confirmed-fraud reports as fast, partial labels to react sooner, and monitor leading indicators (approval-rate and score-distribution shifts) that move before chargebacks confirm them.
| Approach | What you optimize | Why it fails / works for fraud |
|---|---|---|
| Accuracy, threshold 0.5 | Correct-classification rate | Approve-everything scores 99%+; useless |
| Maximize AUC / F1 | Ranking / balanced error | Ignores that a $5k miss and a $5 miss are not equal |
| Expected dollar loss | Money lost to fraud + declines | Right objective: amount-aware, business-aligned |
| Capacity-bounded threshold | Loss subject to review headcount | Realistic: gray zone can only be as big as analysts can clear |
Imbalance and metrics. Do not chase accuracy; with a ~0.5% positive rate it is meaningless. Use PR-AUC (precision-recall, which focuses on the rare positive class), report precision at a fixed recall and recall at a fixed precision, and ultimately translate to dollars caught vs dollars of good transactions blocked. Handle the imbalance in training with class weighting or focal loss rather than blind oversampling, which can manufacture unrealistic synthetic fraud. The crucial operational point: the review queue has fixed capacity. If analysts can clear 5,000 cases a day, the gray-zone threshold must produce about 5,000 cases a day, so the operating point is chosen by precision at the volume the team can actually handle, not by the F1 optimum on a chart.
Adversarial drift makes this unlike a stationary problem. Fraudsters probe your system, find what gets approved, and pour volume through that gap, so your distribution shifts because you deployed a model. Defenses: monitor score distributions and feature drift continuously, retrain frequently (with the label-delay caveat), keep a fast-reacting rules layer on top of the model for emerging patterns analysts spot before the next retrain, and run champion/challenger so a new model is validated on live traffic before it owns the decision. Watch for a feedback loop blind spot: you never see the outcome of transactions you blocked (no chargeback can occur on a declined charge), so the model can never learn it was wrong to block them. Counter it by approving a small randomized holdout through the gray zone to keep an unbiased signal on what you are declining.
Online evaluation and guardrails. A/B (or champion/challenger) on the metrics that matter: fraud dollars caught, false-decline rate on good customers, and queue volume versus capacity. Guardrails that should auto-rollback: a false-decline spike (you are now declining good customers and torching revenue and trust), a review queue blowing past capacity, or an approval-rate cliff in any segment. Bias matters here too: a model can decline disproportionately by geography or card type, so monitor decline rates by segment, since "more fraud caught" is not a win if it is built on declining a legitimate region.
What interviewers probe next
- "Your accuracy is 99.5%, ship it?" No: at sub-1% fraud, approve-everything already hits that. Show PR-AUC and the dollars-caught vs dollars-falsely-declined tradeoff at the chosen operating point.
- "How do you handle that chargebacks arrive weeks late?" Treat label maturity explicitly, use analyst confirmations as fast partial labels, and monitor leading indicators (score and approval-rate drift) that move before chargebacks confirm a new attack.
- "Where do you set the threshold?" Where expected dollar loss is minimized subject to review capacity: the gray zone can only be as large as analysts can clear, so precision-at-volume sets the cutoff, not the F1 peak.
- "Fraudsters adapt to your model." Continuous drift monitoring, frequent retrains, a rules layer for fresh patterns, champion/challenger before promotion, and a randomized approve-through holdout so you still learn about the transactions you block.
Common mistakes
Optimizing accuracy (or picking a 0.5 cutoff) on a 0.5%-positive problem, which yields an approve-everything model. Treating a $5 and a $5,000 transaction with the same probability identically instead of thresholding on dollars. Assuming labels are immediate and reporting glowing metrics on recent data whose fraud has not yet charged back. Choosing the operating point from an F1 chart while the review team can clear a tenth of the resulting cases. Forgetting the adversary, so the model is static while fraud routes around it. And ignoring the blocked-transaction blind spot, so the model never learns which declines were mistakes.
Key takeaways
- Threshold on expected dollar loss (p(fraud) times amount vs the cost of a false decline), never on accuracy or a fixed probability; the amount is part of the decision.
- Two tiers: a ~50ms real-time scorer that splits approve / block / gray zone, plus an async review queue whose capacity caps the gray-zone threshold.
- Velocity and graph features beat raw fields; serve them from a feature store to avoid train/serve skew.
- Plan around label delay (chargebacks land weeks later), adversarial drift, and the blocked-transaction blind spot; evaluate in dollars and false-declines with auto-rollback guardrails.