Activation Functions
Without a non-linear activation, stacking layers is pointless: the whole network collapses into one linear map. Sigmoid and tanh saturate and kill gradients in deep nets, ReLU fixed that but invented dead neurons, and LeakyReLU, GELU and SiLU patch the dead-neuron problem. Softmax normalizes class scores and attention weights.
Transcript
Stack as many linear layers as you like. You've still built one linear layer. The activation function is the entire reason deep networks have depth.
A linear layer computes W x plus b. Stack two and multiply it out. You get another linear map, one that can only draw straight decision boundaries however many layers you pile on. Insert a non-linearity between the layers and each one can bend the space, so the network can approximate arbitrary curved functions.
Sigmoid squashes any input into the range zero to one, and tanh, its zero-centered cousin, maps it from minus one to one; both were the default in early neural nets. Both have a fatal flaw. They saturate: for large positive or negative inputs the curve flattens, so its derivative goes to nearly zero.
Backpropagation multiplies gradients layer by layer. Sigmoid's derivative peaks at just 0.25, and out in the tails it's almost nothing, so every layer shrinks the gradient again and the product falls off exponentially as it flows backward toward the input. The early layers stop learning.
ReLU, the max of zero and x, broke the logjam. For positive inputs its derivative is exactly one, so gradients pass through unscaled. It's also dirt cheap. Its weakness is the flat half. For any negative input it outputs zero, and its gradient is zero. If a neuron's weights drift so it always gets negative inputs, it outputs zero forever, with no gradient to recover, and a large learning rate can kill a sizable fraction of a layer this way.
LeakyReLU patches that with a small slope on the negative side, 0.01 x instead of zero, so a neuron always has some gradient to climb back from. GELU and SiLU are the smooth modern choices, especially in transformers. Both are smooth everywhere, with no hard kink at zero, and both allow a small negative output. The price is compute: they cost more than ReLU, which is why ReLU still wins where speed matters most.
Softmax is different. It turns a vector of raw scores into probabilities that sum to one. Larger scores get more probability. For a multiclass classifier, use it at the output with cross-entropy. Don't replace every hidden activation with it: the features would compete for a fixed total. But attention uses softmax internally to turn query-key scores into weights.
Interviewers want the activation tied to a training symptom. A deep sigmoid net that won't learn? Vanishing gradients. A layer where a chunk of neurons output constant zero points at dead ReLU units, from a learning rate that's too high. Lower the learning rate, or switch to LeakyReLU or GELU.
End with a default. ReLU for general hidden layers, GELU or SiLU inside transformers, and softmax for class probabilities or attention weights. The table and the code are on fdeinterviews.com.
TL;DR: Activations inject the non-linearity that lets a deep net learn curves; without them, any stack of layers is just one linear layer. Sigmoid and tanh saturate at the extremes and cause vanishing gradients in deep models, so ReLU became the default for hidden layers, at the cost of dead neurons. LeakyReLU, GELU and SiLU smooth that out. Use softmax for class probabilities at a classifier's output; attention also uses it internally to normalize scores.
Why you need non-linearity at all
A linear layer computes Wx + b. Stack two of them and you get W2(W1 x + b1) + b2, which is still just W' x + b', another linear map. No matter how many linear layers you stack, the whole thing collapses into a single linear function, which can only draw straight decision boundaries. The activation function is what breaks that collapse. Insert a non-linearity between layers and each layer can bend the space, so the network can approximate arbitrary curved functions. This is the entire reason deep networks have depth.
Sigmoid, tanh, and vanishing gradients
Sigmoid squashes any input into (0, 1) with 1 / (1 + e^-x). Tanh is its zero-centered cousin, mapping to (-1, 1). Both were the default in early neural nets, and both have a fatal flaw in deep models: they saturate. For large positive or negative inputs, the curve flattens, so its derivative goes to nearly zero.
That matters because backpropagation multiplies gradients layer by layer. When each layer's activation contributes a derivative below 1 (sigmoid's max derivative is just 0.25), the product shrinks exponentially as it flows backward. By the time the gradient reaches the early layers, it is effectively zero and those layers stop learning. This is the vanishing gradient problem, and it is why training deep sigmoid networks was so painful. Tanh is somewhat better because it is zero-centered (which keeps gradient signs balanced), but it still saturates.
ReLU and dead neurons
ReLU, max(0, x), broke the logjam. For positive inputs its derivative is exactly 1, so gradients pass through unscaled and do not vanish through depth. It is also dirt cheap to compute. This single change made very deep networks trainable and is why ReLU became the default hidden activation.
Its weakness is the flat half. For any negative input ReLU outputs zero and its gradient is zero. If a neuron's weights drift so that it always receives negative inputs, it outputs zero forever and gets no gradient to recover. It is a dead neuron, permanently stuck, and a large learning rate can kill a sizable fraction of a layer this way.
import numpy as np
def relu(x): return np.maximum(0, x)
def leaky(x, a=0.01): return np.where(x > 0, x, a * x)
def sigmoid(x): return 1 / (1 + np.exp(-x))
def silu(x): return x * sigmoid(x) # SiLU / swish
x = np.array([-3.0, -0.5, 0.0, 0.5, 3.0])
print("relu ", relu(x)) # negatives -> 0 (the dead-neuron risk)
print("leaky", leaky(x)) # negatives keep a small slope
print("silu ", np.round(silu(x), 3)) # smooth, slightly negative dip
LeakyReLU, GELU, SiLU
LeakyReLU patches the dead-neuron problem by giving the negative side a small slope (0.01x instead of 0), so a neuron always has some gradient to climb back from. PReLU makes that slope learnable.
GELU and SiLU (also called swish) are the smooth modern choices, especially in transformers. GELU weights the input by the probability it would survive under a Gaussian gate; SiLU is x * sigmoid(x). Both are smooth everywhere (no hard kink at zero) and allow a small negative output, which empirically trains a bit better in large models. They cost more to compute than ReLU, which is why ReLU still wins where speed matters most.
Softmax is for outputs
Softmax turns a vector of raw scores into probabilities that sum to 1. Larger scores get more probability. For a multiclass classifier, use it at the output with cross-entropy. That does not guarantee calibrated confidence; check calibration separately. In an ordinary dense hidden layer, replacing every activation with softmax makes the features compete for a fixed total. Attention uses softmax internally to turn query-key scores into weights, where that normalization is exactly what is needed. See the scaled dot-product attention equation in Attention Is All You Need.
| Activation | Output range | Pros / cons |
|---|---|---|
| Sigmoid | (0, 1) | Smooth, probabilistic; saturates, vanishing gradients, not zero-centered |
| Tanh | (-1, 1) | Zero-centered, better than sigmoid; still saturates |
| ReLU | [0, inf) | Cheap, no vanishing gradient on positive side; dead neurons |
| LeakyReLU | (-inf, inf) | Fixes dead neurons with a small negative slope; extra hyperparameter |
| GELU / SiLU | ~(-0.3, inf) | Smooth, strong in transformers; pricier than ReLU |
| Softmax | (0, 1), sums to 1 | Class probabilities at the output; attention weights internally |
Why interviewers probe this
The fast filter is whether you can explain why non-linearity is required at all (the linear collapse argument) without hand-waving. The deeper screen is whether you connect activation choice to a training symptom: a deep sigmoid net that will not learn points at vanishing gradients, and a layer where a chunk of neurons output constant zero points at dead ReLUs from too high a learning rate. A strong answer ends with a default: ReLU for general hidden layers, GELU or SiLU inside transformers, softmax for class probabilities or attention weights.
Common misconceptions
- "More layers always means more expressive." Not without non-linear activations between them; linear layers collapse into one.
- "ReLU has no downsides." Dead neurons. Watch for layers stuck at zero and lower the learning rate or switch to LeakyReLU/GELU.
- "Sigmoid is fine for hidden layers." In deep nets it saturates and vanishes the gradient; reserve it for binary output or gates.
- "Softmax is just another activation." It normalizes a vector jointly, making its entries compete for a fixed total. Use that deliberately for class probabilities or attention weights, rather than replacing every dense hidden activation with it.
Key takeaways
- Activations supply non-linearity; without them a deep network is mathematically just one linear layer.
- Sigmoid and tanh saturate and cause vanishing gradients through depth; that is why ReLU replaced them in hidden layers.
- ReLU is cheap and gradient-friendly on the positive side but creates dead neurons; LeakyReLU, GELU and SiLU fix that.
- Use softmax with cross-entropy for multiclass outputs, and within attention to normalize scores. Softmax probabilities still need a separate calibration check.