FDEInterviews logo
← Machine Learning & Data Science / 05
mediumScaleGoogleMicrosoft

The customer's dataset is 99.5% negatives. Their model shows 99.5% accuracy. Walk me through what you'd do.

The accuracy trap is the easy part. Interviewers keep pushing: resample or reweight? Does SMOTE survive contact with production? What happens to your probabilities? Here's the full chain.

Updated Sep 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.

TL;DR: 99.5% accuracy means the model predicts "negative" for everything. Fix measurement (precision/recall, PR AUC), then training (class weights beat SMOTE in production), then the operating point, and recalibrate, because resampling distorts the probabilities every downstream dollar threshold depends on.

Imbalanced: 95% negative, 5% positive positivenegative Oversample duplicate / SMOTEthe minority Undersample drop most of themajority Reweight keep counts, weightminority errors more First fix the metric (PR-AUC, recall); resampling and weights are the levers.

How to approach it

Name the trap immediately, predicting "negative" for everything scores 99.5% accuracy and catches zero events, then structure the answer in three moves: fix the measurement, fix the training, fix the operating point.

A strong answer

Measurement first.

99.5% accuracy on 99.5% negatives 1 The claim 99.5% accuracy, they say 2 Always say negative and you score exactly that 3 Fix measurement first precision, recall, PR AUC 4 Build the eval set stratified, before the model 5 Class weights no data distortion 6 Or get more positives often the highest leverage 7 Pick an operating point the business threshold 8 Recalibrate resampling moved them At 0.5% positives a ten-thousand-row validation set holds about fifty positive examples, so every metric estimate is noisy. Stratify the split and pool more history just for evaluation. One line, works with logistic regression and tree ensembles, and on tabular problems it typically matches or beats SMOTE. The production-safe default. SMOTE interpolates in feature space and routinely manufactures impossible records: a synthetic transaction halfway between two real ones can violate a business rule outright. Every downstream dollar threshold depends on the probability, and resampling distorts it. Skipping this is how a good ranker becomes a bad decision.

Step 2 is the whole diagnosis and it takes one sentence. Everything after it is ordered by how much damage it can do: measurement changes nothing about the data, and step 8 is what keeps a better ranker from becoming a worse decision. Switch to precision/recall, PR AUC, and a confusion matrix at a business-relevant threshold. Build the evaluation set before touching the model, with 0.5% positives, a 10,000-row validation set holds only ~50 positive examples, so metric estimates are noisy; stratify the split and consider pooling more history just for evaluation.

Training fixes, in order of preference:

  1. Class weights / cost-sensitive loss, weight the minority class higher (e.g., class_weight={0:1, 1:50}). No data distortion, one-line change, works with logistic regression and tree ensembles. This is the production-safe default.
  2. Undersample the majority, train on all ~500 positives plus a sample of negatives. Fast iterations; you discard information but with 100k+ negatives that's usually fine.
  3. Oversampling / SMOTE, duplicate or synthesize minority examples. Be openly skeptical: SMOTE interpolates in feature space and routinely manufactures impossible records with real customer data (a synthetic "transaction" halfway between two real ones may violate business rules), and on tabular problems class weights typically match or beat it. Saying "I rarely need SMOTE in production" reads like someone who has shipped.
  4. Get more positives, often the highest-leverage move an FDE can make: mine another year of history, broaden the positive definition with the customer, or commission labeling (this is Scale's home turf). 500 → 5,000 real positives beats any resampling trick.

Operating point last. Resampling and reweighting distort predicted probabilities, and the distortion is computable, which is worth doing once so the customer conversation has numbers in it. Train on data rebalanced to 50/50 from a true base rate of 0.5%, and a score of "60% fraud" from the balanced model corresponds, after correcting the prior shift back, to a true probability of about 0.75%. A "90% confident" alert maps back to about 4.3% (both computed with the standard prior-correction odds adjustment). That is the exact anatomy of the classic complaint, "every alert says 90% and only a few percent are real": nobody's model was broken, the probabilities were simply speaking the rebalanced world's language while the thresholds and dollar rules listened in the real world's. Either recalibrate (Platt scaling / isotonic regression on an unbalanced holdout) or pick thresholds empirically on real-distribution data. Then set the threshold against the customer's constraint: "your team can review 200 alerts/day; here's the recall that buys."

What interviewers probe next

  • "Why do class weights and oversampling behave similarly?", both reshape the effective loss; weights do it without inflating dataset size or leaking duplicates across CV folds.
  • "Where does SMOTE leak?", if you oversample before splitting, synthetic points derived from training rows land in validation. Resample inside each fold only.
  • "Extreme imbalance, 1 in 100,000?", consider anomaly-detection framing (train on negatives only), two-stage funnels (cheap high-recall filter, then a precise model), and active learning to label the most informative cases.
  • "How do you explain the probability distortion to a customer?", show calibration curves; promise alert rankings are trustworthy even while raw scores are being recalibrated.

Common mistakes

Stopping at "accuracy is misleading, use F1." Reaching for SMOTE as a reflex because it's the famous answer, interviewers at Scale and Google increasingly probe whether you know its production failure modes. Resampling before the train/test split (leakage). And forgetting calibration entirely, which surfaces weeks later when the customer asks why every alert claims 90% confidence and only 3% are real.

That one was free — and so are 10 answers per topic without an account. Signing in doubles that to 20, opens the Plus lessons in the courses, and remembers which topics you keep getting wrong.no card · Google sign-in · nothing to cancel
HOW DID IT GO?
0
READING SIGNED OUT

Signing in doubles your free answers, from 10 to 20 per topic, and the site starts remembering you: mastery per topic, bookmarks, and a next-focus recommendation. Free, no card.

Sign in free
UP NEXT ON YOUR JOURNEY
Next in this trackYour model scored 95% in the pilot and 70% in production. What happened?Next in this trackExplain k-fold cross-validation, and when would you refuse to use it?Next in this trackA customer says 'we want the model to be accurate.' How do you turn that into the metric you'll optimize?
FEDITOR'S NOTE

The probability question is the trap that catches people who reach for resampling first: oversampling or class weights distort the base rate, so your predicted probabilities no longer mean what a calibrated probability should, and any downstream dollar threshold breaks. The senior answer is that class weights usually beat SMOTE in production because synthetic minority points often interpolate across the decision boundary, and that you recalibrate afterward rather than trusting the shifted scores.

DISCUSSION · 0

No comments yet — be the first to share your approach.