Loss Functions

TL;DR: The loss is the single number gradient descent minimizes, so it defines what "good" means for your model. Pick MSE when large errors should be punished hard, MAE or Huber when outliers shouldn't dominate, cross-entropy for classification because it pairs with probabilities, and contrastive or triplet losses to shape embedding geometry. The mistake that fails interviews: optimizing a loss that does not match the metric the business grades you on.

What the loss actually does

A model has weights, and training nudges them downhill on some scalar that measures wrongness. That scalar is the loss. Everything else (the optimizer, the learning rate, the schedule) only moves you down the surface the loss defines. So the loss is not a detail you pick last; it is the shape of the problem. Change the loss and you change which mistakes the model is willing to make.

Two things matter when choosing one: what it optimizes (what kind of error it cares about) and how its gradient behaves (whether it gives the optimizer a clean signal to follow).

Regression losses: MSE, MAE, Huber

For predicting a continuous number, the three workhorses differ entirely in how they treat large errors.

Mean squared error, mean((y - yhat)^2), squares each error. A prediction off by 10 contributes 100 times more than one off by 1, so MSE is dominated by outliers and pulls the fit toward them. Its gradient is proportional to the error, which is convenient: big mistakes produce big corrections, and it shrinks smoothly to zero near the optimum.

Mean absolute error, mean(|y - yhat|), treats an error of 10 as exactly ten times an error of 1. Its penalty grows linearly, so large errors have less influence than under MSE; the penalty is still unbounded. The cost is its gradient: it is constant (sign of the error) everywhere except a kink at zero, so the optimizer takes the same size step whether it is far off or nearly right, which makes the final convergence jittery.

Huber loss is the pragmatic merge. It behaves like MSE for small residuals (smooth gradient near the optimum) and like MAE for large ones (a linear tail with a bounded gradient), switching at a tunable threshold delta.

import numpy as np

def mse(y, yhat):  return np.mean((y - yhat) ** 2)
def mae(y, yhat):  return np.mean(np.abs(y - yhat))

def huber(y, yhat, delta=1.0):
    e = np.abs(y - yhat)
    quad = np.minimum(e, delta)
    lin = e - quad
    return np.mean(0.5 * quad ** 2 + delta * lin)

y    = np.array([10.0, 12.0, 11.0, 50.0])   # 50 is an outlier
yhat = np.array([10.0, 12.0, 11.0, 13.0])
print(mse(y, yhat), mae(y, yhat), huber(y, yhat))  # MSE blows up; MAE/Huber stay sane

342.25 9.25 9.125

Executed. Three predictions are exact and one is off by 37, and MSE reports 342 while MAE reports 9.25: the single outlier is the whole MSE (37 squared, divided by four) and the whole MAE too (37 divided by four). Huber with a delta of 1 charges the outlier a linear 36.5 instead of a squared 1,369, which is why its 9.1 sits beside MAE rather than MSE. If that fourth point is a data-entry error, MAE and Huber give it less influence than MSE. You still need to investigate the point; a small loss does not establish that the model is fine.

Classification: cross-entropy

Cross-entropy trains a classifier's predicted probabilities. Minimizing it does not guarantee calibrated confidence on held-out data: modern neural networks can remain overconfident, so check calibration separately. For a true class, the loss is -log(p) where p is the probability the model assigned to the correct class. Predict the right class with high confidence and the loss is near zero; assign the correct class a probability near zero and the loss explodes toward infinity. That asymmetry is the point: confidently wrong is punished savagely, which is exactly what you want from a probability model.

Cross-entropy is not an arbitrary choice. It is the negative log-likelihood under the model, and it falls straight out of information theory: minimizing it minimizes the extra bits needed to encode the true labels using the model's predicted distribution. Paired with a softmax output (multiclass) or sigmoid (binary), its gradient simplifies to predicted - true, a clean, well-behaved signal that does not saturate the way squared error on probabilities does. In numbers: with the correct class at a predicted 0.01, the cross-entropy gradient on the logit is -0.99, a hard push; squared error on the probability carries an extra factor of p(1 - p) and gives -0.02 at the same point, a nudge, so the worse the prediction, the less squared error does about it (computed). At 0.5 the two are -0.5 and -0.25; only cross-entropy keeps pulling hardest where the model is most wrong.

Hinge and embedding losses

Hinge loss, max(0, 1 - y * score), is the SVM objective. It penalizes every prediction that violates the margin, including confidently wrong predictions far from the boundary. Once a point is correctly classified with margin to spare, it contributes zero gradient. It optimizes a decision boundary. Cross-entropy trains probabilities, but you still need to check calibration.

Contrastive and triplet losses shape an embedding space rather than predict a label. Triplet loss takes an anchor, a positive (same class) and a negative (different class), and pushes the anchor closer to the positive than to the negative by a margin: max(0, d(a,p) - d(a,n) + margin). This is how face recognition and retrieval embeddings are trained, where the goal is geometry (similar things are near each other) rather than a class.

Pick the loss to match the metric

The recurring trap is a mismatch between the loss and the metric you actually care about. Compare two constant predictors on a 99-to-1 split: a model that says 0.01 for the rare class on every input has a mean cross-entropy of 0.056, against 0.693 for one that hedges at 0.5 (computed). The first wins this comparison, yet catches no rare cases at a 0.5 decision threshold. This compares constant predictors; informative features can support a better model. Evaluate recall and the decision threshold, rather than inferring them from loss alone. Class weights or focal loss can change the training objective when the costs of mistakes differ.

Why interviewers probe this

Naming MSE and cross-entropy is table stakes. The screen is whether you connect the loss to the gradient and to the metric. A strong answer explains why cross-entropy beats squared error for classification (probability training, a non-saturating predicted - true gradient) and catches the loss-metric mismatch when the interviewer describes an imbalanced or asymmetric-cost problem. The follow-up they hold in reserve is usually "your accuracy is great but the rare class is never caught, why?" and the answer should check the decision threshold, the available signal and whether the objective reflects the costs of mistakes.

Common misconceptions

• "Use MSE for everything numeric." MSE is dominated by outliers. If a few extreme points should not steer the model, use MAE or Huber.
• "Cross-entropy and accuracy are the same goal." Cross-entropy scores the probability assigned to the true label; accuracy only cares about the argmax. A model can improve one while flat on the other.
• "The loss and the evaluation metric must be identical." They often cannot be (accuracy is non-differentiable), but the loss must be a faithful proxy. A mismatch is the bug, not the design.
• "Bigger loss values mean a worse model across losses." Loss magnitudes are not comparable across different loss functions or scales; only the trend within one training run is meaningful.

Key takeaways

• The loss defines what the model optimizes; choosing it is choosing which errors are acceptable.
• MSE punishes outliers hard, MAE grows linearly with their magnitude, Huber blends both; pick by how much outliers should steer the fit.
• Cross-entropy is the classification default because it trains probabilities with a clean predicted - true gradient.
• The loss must match the metric you are judged on; for an imbalanced or asymmetric-cost problem, evaluate the decision threshold and consider class weights or focal loss.