Your model scored 95% in the pilot and 70% in production. What happened?

Question

FDEInterviews · Accepted Answer

TL;DR: A 25-point pilot-to-production drop is leakage until proven otherwise: a transform fit on the full dataset, a feature that secretly encodes the future, or the same entity in both splits. Confirm both numbers measure the same thing, then run a ranked differential and ask of every feature "could this value have been known at decision time?" How to approach it Treat it as an incident triage, not a guessing game: "Before hypothesizing, I'd confirm both numbers are computed the same way, same metric, same threshold, same population. Half of these gaps are measurement mismatches." Then walk a ranked differential diagnosis. A strong answer Suspect #1: train/test leakage, information available at training time that won't exist at prediction time. The enterprise classics: Temporal leakage: random splits on time-ordered data. A churn model trained on a random split learns from March to "predict" February. Fix: time-based splits, always, for anything with a timestamp. Feature leakage: a feature that encodes the label. Real example: predicting hospital readmission with a dischargedisposition field that includes "transferred for readmission", 95% offline, useless live. In customer warehouses, watch for fields populated after the outcome (status columns, audit fields updated post-hoc, aggregates computed over the full table including the future). Entity leakage / duplicates: the same customer's records in both train and test. The model memorizes entities, not patterns. Fix: group-aware splits (split by customer ID, not by row). Preprocessing leakage: fitting scalers, encoders, or imputers on the full dataset before splitting. Subtle, ubiquitous, fixed by pipelines that fit only on training folds. How much can preprocessing leakage inflate a score? Enough to manufacture a working model out of nothing, and the experiment fits in twenty lines, so we ran it. Three hundred rows, a 100-value categorical feature, and labels that are pure coin flips, no signal exists by construction. Target-encode the categorical on the full dataset (each category replaced by its mean label, computed over all rows) and then cross-validate: 70% accuracy. On noise. The encoding smuggled each test row's own label into its feature, because that row participated in its category's mean. Fit the same encoding inside each fold instead, on training rows only, and accuracy collapses to chance (44% on this run). That is a 26-point mirage from one innocent-looking line of pandas executed before the split, which is roughly the size of the pilot-to-production gap in this question's title, and why "every transform fits inside the fold" is a rule and not a preference. Suspect #2: training/serving skew, the production feature pipeline computes features differently than the offline one (different null handling, different aggregation windows, a join that times out and silently returns defaults). Diagnose by logging served features and diffing distributions against training data, feature by feature. Suspect #3: distribution shift, the pilot used last year's data; the world moved. Compare population statistics between pilot and production traffic (PSI per feature is a quick screen). The senior move is the audit procedure: take five production predictions that went wrong, pull the exact feature vectors the model saw, and ask of each feature, "could this value have been known at decision time?" That single question catches most leakage. Then state the prevention: point-in-time-correct training data (snapshot features as they were when each label event happened, this is the core problem feature stores exist to solve), time-based evaluation, and a shadow-mode period before the pilot readout so offline and online numbers are reconciled before anyone presents 95% to an exec. What interviewers probe next "How would you detect leakage before deployment?", suspiciously high scores are themselves a signal ("if AUC is 0.99 on a hard business problem, I assume leakage until proven otherwise"); feature importance dominated by one field; performance that collapses on a time split. "What do you tell the customer who already saw 95%?", reset expectations fast and honestly: explain the flaw, present the corrected number, and re-anchor success criteria on the honest baseline. Hiding it for a week is the deal-killer. "Group splits vs time splits, which wins?", whichever matches deployment: predicting for new customers → group split; predicting the future → time split; often both. Common mistakes Jumping to "data drift" because it's blameless, leakage is more common and it's your pipeline's fault. Naming one leakage type instead of the taxonomy. No verification procedure, just vibes. And missing the customer half: an FDE who fixes the model but lets the exec keep quoting "95%" has set the account up to fail at renewal.