Loss Functions

TL;DR: The loss is the single number gradient descent minimizes, so it defines what "good" means for your model. Pick MSE when large errors should be punished hard, MAE or Huber when outliers shouldn't dominate, cross-entropy for classification because it pairs with probabilities, and contrastive or triplet losses to shape embedding geometry. The mistake that fails interviews: optimizing a loss that does not match the metric the business grades you on.

What the loss actually does

A model has weights, and training nudges them downhill on some scalar that measures wrongness. That scalar is the loss. Everything else (the optimizer, the learning rate, the schedule) only moves you down the surface the loss defines. So the loss is not a detail you pick last; it is the shape of the problem. Change the loss and you change which mistakes the model is willing to make.

Two things matter when choosing one: what it optimizes (what kind of error it cares about) and how its gradient behaves (whether it gives the optimizer a clean signal to follow).

Regression losses: MSE, MAE, Huber

For predicting a continuous number, the three workhorses differ entirely in how they treat large errors.

Mean squared error, mean((y - yhat)^2), squares each error. A prediction off by 10 contributes 100 times more than one off by 1, so MSE is dominated by outliers and pulls the fit toward them. Its gradient is proportional to the error, which is convenient: big mistakes produce big corrections, and it shrinks smoothly to zero near the optimum.

Mean absolute error, mean(|y - yhat|), treats an error of 10 as exactly ten times an error of 1. It is robust to outliers because no single point can blow up. The cost is its gradient: it is constant (sign of the error) everywhere except a kink at zero, so the optimizer takes the same size step whether it is far off or nearly right, which makes the final convergence jittery.

Huber loss is the pragmatic merge. It behaves like MSE for small residuals (smooth gradient near the optimum) and like MAE for large ones (a linear, bounded penalty so outliers do not dominate), switching at a tunable threshold delta.

import numpy as np

def mse(y, yhat):  return np.mean((y - yhat) ** 2)
def mae(y, yhat):  return np.mean(np.abs(y - yhat))

def huber(y, yhat, delta=1.0):
    e = np.abs(y - yhat)
    quad = np.minimum(e, delta)
    lin = e - quad
    return np.mean(0.5 * quad ** 2 + delta * lin)

y    = np.array([10.0, 12.0, 11.0, 50.0])   # 50 is an outlier
yhat = np.array([10.0, 12.0, 11.0, 13.0])
print(mse(y, yhat), mae(y, yhat), huber(y, yhat))  # MSE blows up; MAE/Huber stay sane

Classification: cross-entropy

For classification you want calibrated probabilities, not raw scores, and cross-entropy is the loss that rewards them. For a true class, the loss is -log(p) where p is the probability the model assigned to the correct class. Predict the right class with high confidence and the loss is near zero; assign the correct class a probability near zero and the loss explodes toward infinity. That asymmetry is the point: confidently wrong is punished savagely, which is exactly what you want from a probability model.

Cross-entropy is not an arbitrary choice. It is the negative log-likelihood under the model, and it falls straight out of information theory: minimizing it minimizes the extra bits needed to encode the true labels using the model's predicted distribution. Paired with a softmax output (multiclass) or sigmoid (binary), its gradient simplifies to predicted - true, a clean, well-behaved signal that does not saturate the way squared error on probabilities does.

Hinge and embedding losses

Hinge loss, max(0, 1 - y * score), is the SVM objective. It only penalizes predictions inside a margin of the boundary; once a point is correctly classified with margin to spare, it contributes zero gradient. That makes it focus capacity on the hard, near-boundary cases. It optimizes for a decision boundary, not a probability, so reach for cross-entropy if you need calibrated confidence.

Contrastive and triplet losses shape an embedding space rather than predict a label. Triplet loss takes an anchor, a positive (same class) and a negative (different class), and pushes the anchor closer to the positive than to the negative by a margin: max(0, d(a,p) - d(a,n) + margin). This is how face recognition and retrieval embeddings are trained, where the goal is geometry (similar things are near each other) rather than a class.

Pick the loss to match the metric

The recurring trap is a mismatch between the loss and the metric you actually care about. If the business grades you on recall of a rare class, plain cross-entropy on a 99-to-1 split will happily learn to predict the majority class, because that genuinely minimizes the loss. You fix it by changing the objective (class weights, focal loss) so the loss agrees with the metric, not by training harder on the wrong one.

Why interviewers probe this

Naming MSE and cross-entropy is table stakes. The screen is whether you connect the loss to the gradient and to the metric. A strong answer explains why cross-entropy beats squared error for classification (calibrated probabilities, a non-saturating predicted - true gradient) and catches the loss-metric mismatch when the interviewer describes an imbalanced or asymmetric-cost problem. The follow-up they hold in reserve is usually "your accuracy is great but the rare class is never caught, why?" and the answer is that the loss never asked the model to catch it.

Common misconceptions

• "Use MSE for everything numeric." MSE is dominated by outliers. If a few extreme points should not steer the model, use MAE or Huber.
• "Cross-entropy and accuracy are the same goal." Cross-entropy rewards calibrated probability; accuracy only cares about the argmax. A model can improve one while flat on the other.
• "The loss and the evaluation metric must be identical." They often cannot be (accuracy is non-differentiable), but the loss must be a faithful proxy. A mismatch is the bug, not the design.
• "Bigger loss values mean a worse model across losses." Loss magnitudes are not comparable across different loss functions or scales; only the trend within one training run is meaningful.

Key takeaways

• The loss defines what the model optimizes; choosing it is choosing which errors are acceptable.
• MSE punishes outliers hard, MAE ignores their magnitude, Huber blends both; pick by how much outliers should steer the fit.
• Cross-entropy is the classification default because it yields calibrated probabilities and a clean predicted - true gradient.
• The loss must match the metric you are judged on; an imbalanced or asymmetric-cost problem needs class weights or a margin loss, not just more training.