TL;DR: Detect overfitting as a train-versus-validation gap, and add a time-split check so you catch the model that aces a random split but fails the future. Then work the fix toolbox in order: more or cleaner data first, then fewer features, then regularization and early stopping, not the reverse.
How to approach it
Lead with detection, because in customer settings overfitting is usually discovered too late: "Detection is a gap, strong performance on training data, weak on held-out data. My job is to surface that gap before the customer's users do."
A strong answer
Detection, concretely:
Steps 4 to 7 are deliberately in that order. Most people reach for step 6 first because it feels like modeling, and it is the weakest of the four. split data into train/validation/test (say 70/15/15), and watch the curves during training. Training loss falling while validation loss rises is the signature. In deployment work, add a second detector: evaluate on a time-split, train on January–April, validate on May. A model that's fine on a random split but bad on a time split is overfitting to a moment in time, which is exactly how it will fail in production.
Then the fix toolbox, in the order you'd actually try it:
- More/cleaner data, the most reliable fix, and in customer work often available: "you have 18 more months of history in that other system; let's pull it."
- Simpler model or fewer features, drop from 200 features to the 30 that matter; shallower trees; fewer boosting rounds.
- Regularization, L2 (ridge) shrinks all weights smoothly, fighting variance while keeping every feature; L1 (lasso) drives weak weights to exactly zero, doing feature selection for free, useful when the customer asks "which inputs does this thing actually use?" For trees: max depth, min samples per leaf, subsampling. For neural nets: dropout, weight decay.
- Early stopping, halt when validation loss stops improving; arguably the most-used regularizer in practice because it's free.
- Cross-validation for trustworthy estimates when data is small (see the CV question later in this track).
Micro-example: a churn model with 5,000 customers and 400 features hits 99% train / 71% validation accuracy. Cutting to 40 features and adding L2 moves it to 88% train / 84% validation, the train number got worse and the model got better. Telling a customer "training accuracy dropped and that's good news" is a genuine FDE moment; rehearse it.
The "can you overfit the validation set?" probe below has a five-line demonstration behind it that is worth having run once. Generate 50 models that are literally coin flips, no signal at all by construction, and score each against a 200-example validation set: the best of the 50 scores 58% (we ran this; the spread comes from binomial noise, roughly plus or minus 3.5% per model, and picking the max harvests the lucky tail). Report that winner and you have manufactured 8 points of accuracy out of pure selection; its true skill is exactly 50%, and the untouched test set is what reveals it. Now replace "50 coin flips" with "50 hyperparameter configs evaluated against the same small validation set" and the arithmetic is identical, which is why the test set stays in a vault, why tiny eval sets make every tuning decision noisy, and why a pilot's final numbers must come from an evaluation period nobody iterated against.
What interviewers probe next
- "L1 vs L2, when each?", L1 when you suspect few features matter or need a sparse, explainable model; L2 as the default when many features carry small signal; elastic net when correlated features make L1's picks unstable.
- "How do you choose lambda?", cross-validated grid/Bayesian search, tuned on validation, never on test.
- "Can you overfit the validation set?", yes, by iterating against it dozens of times. That's why the test set stays in a vault until the end, and why pilots need a final untouched evaluation period.
- "Overfitting in non-model places?", prompt tuning against a tiny eval set, feature engineering using full-dataset statistics, threshold-picking on test data. Naming these earns range.
Common mistakes
Jumping straight to "add dropout" without describing detection. Presenting regularization as the only tool when data and simplicity usually win first. Confusing which norm sparsifies (it's L1). And the deployment-grade miss: validating only on random splits, so the model that aced the pilot quietly decays the moment the future stops resembling the training window.
Key takeaways
- Detection comes before fixes: a widening train-validation gap, confirmed on a time split, is the signature.
- Order the toolbox by reliability: data and feature reduction beat fancy penalties; early stopping is the free default.
- A training accuracy that drops while validation rises is good news; rehearse saying that to a customer.