Overfitting and Regularization

TL;DR: A model overfits when it memorizes the quirks of the training set rather than learning the underlying pattern, so train error keeps dropping while validation error climbs. You spot it from the train-validation gap and you fight it with more or better data, L1/L2 regularization, early stopping, dropout, and cross-validation. The trap is a model that scores 99% on a held-out set yet fails in production because the held-out set was not representative.

What overfitting actually is

Every training set is signal plus noise. The signal is the relationship you want the model to learn. The noise is the accidental stuff: a measurement error, a coincidence in this particular sample, a label that happens to correlate with something irrelevant. A model with enough capacity will happily fit both, because fitting the noise lowers training error just like fitting the signal does. The problem is that the noise does not repeat in new data. So the memorized quirks become dead weight that drags down every future prediction.

The clean way to see it: an overfit model has low bias and high variance. It is flexible enough to match the truth but so flexible it matched the randomness too.

How you spot it

You hold out data the model never trains on, and you watch two curves as training proceeds.

• Train error keeps falling, smoothly, toward zero.
• Validation error falls for a while, then bottoms out and starts rising.

That divergence is the signature. The point where validation error turns up is roughly where the model stops learning signal and starts memorizing noise. The size of the final gap tells you how badly. A model at 99% train accuracy and 70% validation accuracy is overfitting hard; one at 88% and 86% is fine.

Train error keeps falling while validation error bottoms out and turns up; everything right of that minimum is the overfitting region:

Why it happens

Too much model capacity for the amount of data (a deep network or unpruned tree on a small dataset), too many features relative to examples, training for too many epochs, or leakage where a feature secretly encodes the label. The common thread is that the model has more freedom than the data can constrain.

How you prevent it

The fixes fall into a few families, and the first one beats the rest when you can afford it.

• More and better data. The cleanest cure. A flexible model that sees enough varied examples can no longer memorize them, because there is too much to memorize. Better-quality and more-diverse data often beats any clever regularizer.
• L2 regularization (ridge) adds a penalty proportional to the sum of squared weights, loss + lambda * sum(w^2). It shrinks all weights toward zero, discouraging any single feature from dominating, which smooths the model.
• L1 regularization (lasso) penalizes the sum of absolute weights, loss + lambda * sum(|w|). It drives some weights exactly to zero, so it also does feature selection. Reach for L1 when you suspect many features are useless.
• Early stopping. Stop training at the validation-error minimum instead of running to convergence. Simple and effective for iterative learners.
• Dropout (for neural nets) randomly zeroes a fraction of activations each step, so no neuron can rely on a fixed set of partners. It acts like training an ensemble of subnetworks.
• Cross-validation does not prevent overfitting directly; it gives you an honest, lower-variance estimate of generalization so you tune hyperparameters (like lambda) without fooling yourself on one lucky split.

lambda is the dial on regularization strength. Too small and it does nothing; too large and you swing into underfitting (high bias). You pick it by cross-validation.

The "99% test accuracy but fails in production" trap

The most expensive overfitting is invisible on your metrics. Your test accuracy is 99% and the model still fails the day it ships. Two usual causes. First, leakage: a feature available at train time encodes the answer but is absent or different at inference, so the model learned a shortcut that does not exist in production. Second, a held-out set that is not representative of real traffic, so you overfit to a distribution your users do not match. Both produce gorgeous offline numbers and a broken product. The defense is a golden eval set drawn from real usage, refreshed as traffic drifts, and a hard look for any feature that is suspiciously predictive.

Why interviewers probe this

Overfitting is the failure mode behind most "the model worked in the notebook but not in prod" stories, so the loop wants to see that you treat the validation set as sacred and reason about leakage. The follow-up they hold in reserve is "your test accuracy is 99%, are you happy?" The wrong answer celebrates. The strong answer gets suspicious, asks how the test set was built and whether any feature could be leaking the label, and proposes a fresh held-out set from production traffic before trusting the number.

Common misconceptions

• "Regularization always improves the model." Only up to a point. Too much regularization underfits. lambda is a tuned hyperparameter, not a free win.
• "High test accuracy means it will work in production." Not if the test set is unrepresentative or a feature is leaking. Offline metrics are only as honest as the data behind them.
• "L1 and L2 do the same thing." L2 shrinks weights smoothly; L1 zeroes some out and does feature selection. Different tools for different problems.
• "Overfitting only happens with neural networks." Any flexible model overfits: unpruned decision trees, high-degree polynomials, k-NN with k=1. Capacity relative to data is what matters.

Key takeaways

• Overfitting is memorizing noise instead of signal; you see it as a widening gap between low train error and rising validation error.
• The first and best fix is more and better data; regularization, early stopping, and dropout constrain capacity when more data is not available.
• L2 shrinks weights smoothly, L1 zeroes some out and selects features; tune the strength lambda by cross-validation.
• A 99% offline score can still fail in production from leakage or an unrepresentative test set; trust a fresh, representative held-out set, not the convenient one.