Gradient Descent & Learning Rate

TL;DR: Training minimizes a loss function by repeatedly nudging the weights downhill along the gradient (the direction of steepest increase, negated). The learning rate is the step size: too small and you crawl, too large and you overshoot the minimum and can diverge entirely. SGD computes the gradient on small batches for speed; Adam adapts the step per parameter. Most training failures trace back to the learning rate.

The core loop

A model has weights and a loss that measures how wrong it is. The gradient is the vector of partial derivatives of the loss with respect to each weight: it points in the direction that would increase the loss fastest, so the negative gradient points downhill. Gradient descent just walks that way in small steps: compute the gradient, subtract a fraction of it from the weights, repeat. That fraction is the learning rate.

That is the whole idea, and it scales from fitting a line to training a frontier model. What changes is how you estimate the gradient and how you size the step.

The learning rate is the knob that matters

Six steps, and only step 5 has a number you choose. That is why most training failures trace back to it.

Drag the slider above and watch the behavior flip. A tiny learning rate inches toward the bottom and wastes compute. A well-sized rate converges in a few steps. Push it too high and each step overshoots the minimum by more than the last, so the ball climbs the walls instead of settling: the loss diverges to infinity. This is exactly the "loss went to NaN" failure every practitioner has seen, and the first thing to check is almost always the learning rate.

Because a single fixed rate is rarely ideal for the whole run, real training uses a schedule: a brief warmup (start small so early, noisy gradients do not blow up), then a peak, then a decay toward zero so the model settles into a good minimum.

SGD, mini-batches, and Adam

Computing the gradient over the entire dataset every step is too slow, so stochastic gradient descent estimates it on a small random batch. The estimate is noisy, but the noise is cheap and even helps escape bad spots. Momentum smooths the noisy steps by accumulating a running average of past gradients. Adam goes further and keeps a per-parameter adaptive step size, which is why it is the default optimizer for deep learning: it is forgiving about the initial learning rate and handles parameters that need very different step sizes.

Why interviewers probe this

It separates people who have actually trained models from people who have only called .fit(). The tell is whether you reach for the learning rate first when training misbehaves, and whether you can explain warmup and decay rather than reciting "use Adam." A strong answer connects the abstract loop to a real symptom: loss exploding to NaN means the rate is too high, loss barely moving means it is too low or the schedule decayed too early.

Common misconceptions

• "A bigger learning rate trains faster." Up to a point. Past the stability threshold it diverges, and you train nothing at all.
• "Adam means I do not have to tune the learning rate." Adam is forgiving, not immune. The peak rate and schedule still matter.
• "Gradient descent finds the global minimum." In deep nets the loss is non-convex; it finds a good-enough local basin, and that is fine in practice.
• "More steps always help." Past convergence you start fitting noise; this is where it meets overfitting and early stopping.

Key takeaways

• Gradient descent steps weights downhill along the negative gradient; the learning rate is the step size.
• Too small crawls, too large diverges; the NaN loss is the classic too-large symptom.
• Use a schedule (warmup then decay) rather than one fixed rate, and SGD on mini-batches for speed.
• Adam adapts the step per parameter and is the forgiving default, but the rate still needs tuning.