Bias-Variance Tradeoff

TL;DR: Bias is the error from a model too simple to fit the real pattern (it underfits, doing badly on train and validation alike). Variance is the error from a model so flexible it fits the noise (it overfits, acing train and bombing validation). You diagnose which one you have by comparing train error to validation error, then attack that one specifically.

The two ways a model can be wrong

Imagine fitting a curve through scattered data. Draw a straight line and you have made a strong assumption: the world is linear. If it is not, the line misses the shape everywhere, and no amount of new data helps, because the model cannot bend. That systematic miss is bias. The model is too rigid.

Now fit a wiggly high-degree polynomial that threads through every single point. It nails your training data, but the wiggles are chasing random noise, not signal. Show it new data and the predictions swing wildly, because tiny changes in the training sample would have produced a very different curve. That sensitivity is variance. The model is too flexible.

Total expected error on new data decomposes, roughly, into three parts: error = bias^2 + variance + irreducible noise. The noise floor you cannot beat. Bias and variance are the two you control, and they usually pull in opposite directions as you change model complexity. Simpler model: more bias, less variance. More complex model: less bias, more variance. The sweet spot is the complexity that minimizes their sum, not either one alone.

Diagnose with train versus validation error

You do not guess which problem you have. You read it off two numbers.

• High train error, validation error close behind. Both are bad. That is high bias / underfitting. The model cannot even fit the data it has seen, so the fix is a more expressive model, better features, or less regularization.
• Low train error, much higher validation error. A wide gap. That is high variance / overfitting. The model memorized the training set. The fix is more data, regularization, or a simpler model.
• Both low and close. You are in good shape; chase remaining error with features or a different model class.

The gap between train and validation error is the variance signal. The level of train error is the bias signal. Read both at once.

A concrete example

Predicting house prices from square footage. A model using only price = w * sqft + b is a straight line. It will underfit, because price also depends on location, age, and condition, so it misses badly on both train and test. That is bias: add features and capacity.

Swap to a decision tree with no depth limit and one leaf per training house. It predicts the training set perfectly and predicts garbage for any house it has not seen, because it memorized exact addresses rather than learning a relationship. That is variance: limit depth, prune, or move to a random forest that averages many trees to cancel the noise.

The same data, two opposite failures, two opposite fixes. That is why naming which one you have comes before choosing a remedy.

Pushing both down

The trade-off is not a law of nature you must accept; it describes a fixed model and dataset. You can move the whole frontier down:

• More and cleaner data mostly reduces variance: a flexible model that sees enough examples can no longer memorize them.
• Better features reduce bias: they hand the model the signal it was too simple to extract on its own.
• Ensembling (bagging, random forests) reduces variance by averaging many high-variance models whose errors partly cancel.
• Regularization and early stopping trade a little bias for a large drop in variance, which is usually a winning deal.

Why interviewers probe this

This separates people who tune models by feel from people who reason about them. The tell is the diagnosis step. A candidate who says "accuracy is low, let me add layers" without checking the train-validation gap is guessing. The strong move is to state the two numbers first, name bias or variance, then prescribe. The follow-up they hold back is "you added data and nothing improved, now what?" The right answer: if more data did not help, you are bias-limited, not variance-limited, so add capacity or features instead of collecting more rows.

Common misconceptions

• "More data always helps." It mainly cuts variance. If your model is underfitting (high bias), more rows of the same features will not move the needle; you need more expressive models or better features.
• "Bias and variance are about biased data or noisy data." They are properties of the model's fit, not the dataset's fairness or measurement noise. The names are unfortunate.
• "A more complex model is more accurate." Only until variance overtakes the bias you removed. Past the sweet spot, validation error climbs even as train error keeps falling.
• "You must trade one for the other." That holds for a fixed dataset. Better features, more data, and ensembling lower both at once.

Key takeaways

• Bias = too simple, underfits, bad on train and validation alike; variance = too flexible, overfits, great on train and poor on validation.
• Diagnose by reading train error (bias signal) and the train-to-validation gap (variance signal) together before prescribing a fix.
• Total error is roughly bias^2 + variance + irreducible noise; the goal is the complexity that minimizes their sum.
• The trade-off only binds for a fixed model and dataset; more data, better features, and ensembling move the whole curve down.