TL;DR: Gradient descent is repeatedly stepping downhill on the loss landscape, with the learning rate as step size. The deployable skill is reading the loss curve: oscillating or exploding means the rate is too high, flat-from-step-one usually means a data or label bug, and train-down-validation-up is overfitting, not optimization.
How to approach it
This is an explainability test as much as a knowledge test, FDEs explain technical machinery to customer engineers weekly. Give a clean mental model in under two minutes, then show you know the practical failure modes, not the convergence proofs.
A strong answer
The mental model: training a model means finding parameter values that minimize a loss function, a measure of how wrong the model is. Picture the loss as a landscape where altitude is error and your position is the current parameter values. You can't see the whole landscape, but at any point you can feel the slope under your feet (the gradient, computed via backpropagation in neural nets). Gradient descent is: repeatedly step downhill. The step size is the learning rate.
Then the engineering reality, with the failure modes:
- Learning rate too high: you overshoot the valley and bounce between walls, loss oscillates or explodes (NaNs in the logs). Too low: training crawls and can stall on plateaus. This is the single most impactful hyperparameter; in practice you use a schedule (warmup, then decay) and adaptive optimizers like Adam that scale steps per-parameter.
- Batch size: computing the true gradient over 10M rows per step is wasteful, so we estimate it from mini-batches (32–1024 rows). The noise from sampling is a feature, not just a bug, it helps escape shallow local dips and is why it's called stochastic gradient descent.
- Local minima and saddle points: the landscape isn't a single bowl. In high dimensions, saddle points and flat plateaus are the bigger practical problem; momentum (accumulating a velocity vector across steps) rolls through them.
- Feature scaling: if one feature ranges 0–1 and another 0–1,000,000, the landscape becomes a stretched ravine and descent zigzags painfully. Standardizing features makes the bowl round, this is why preprocessing affects optimization, not just statistics.
The three regimes are reproducible on a loss simple enough to run in your head, f(x) = x² starting at x = 10, where each step multiplies x by (1 minus twice the learning rate). We ran ten steps at three rates: at 0.01, x has only crawled to 8.17 and the loss still reads 66.8, the flat line that tempts people to "let it train longer" when the fix is one hyperparameter. At 0.45, x hits 1.0 after a single step and is at the minimum by step five. At 1.05, x goes 10 to -11 to -16.1 and reaches +25.9 by step ten with the loss at 672 and climbing, each step leaping across the valley and landing higher on the far wall, which is exactly what an oscillating-then-NaN training log is showing you at a million parameters instead of one. Same landscape, same algorithm, and the step size alone decides crawl, converge, or explode.
The deployable skill: reading loss curves. Smooth decrease then plateau = healthy. Wild oscillation = learning rate too high. Loss flat from step one = LR too low, bad initialization, or data/label bug (check the data first, at a customer site it's almost always the data). Training loss down, validation loss up = overfitting; stop early.
What interviewers probe next
- "Why does this find good solutions if it's not guaranteed to find the global minimum?", for deep nets, many minima generalize similarly; we want a low, flat basin, not the global optimum. For convex problems (logistic regression), the bowl has one bottom and convergence is guaranteed.
- "Batch vs mini-batch vs single-example SGD?", tradeoff between gradient quality and steps per second; mini-batch wins on GPU parallelism.
- "What's Adam doing, one sentence?", per-parameter learning rates from running averages of gradient mean and variance, momentum plus automatic step scaling.
- "Does any of this matter with XGBoost?", trees don't use gradient descent over parameters, but gradient boosting descends in function space: each tree fits the gradient of the loss. Connecting those reads well.
Common mistakes
Drowning the questioner in calculus when they asked for intuition, failing the customer-communication test inside the ML test. Not knowing what a too-high learning rate looks like in logs. Claiming local minima make training hopeless. And forgetting feature scaling, which is the version of this topic that actually bites during a customer's first training run.