Activation Functions

TL;DR: Activations inject the non-linearity that lets a deep net learn curves; without them, any stack of layers is just one linear layer. Sigmoid and tanh saturate at the extremes and cause vanishing gradients in deep models, so ReLU became the default for hidden layers, at the cost of dead neurons. LeakyReLU, GELU and SiLU smooth that out. Use softmax only on the output of a classifier, never as a hidden activation.

Why you need non-linearity at all

A linear layer computes Wx + b. Stack two of them and you get W2(W1 x + b1) + b2, which is still just W' x + b', another linear map. No matter how many linear layers you stack, the whole thing collapses into a single linear function, which can only draw straight decision boundaries. The activation function is what breaks that collapse. Insert a non-linearity between layers and each layer can bend the space, so the network can approximate arbitrary curved functions. This is the entire reason deep networks have depth.

Sigmoid, tanh, and vanishing gradients

Sigmoid squashes any input into (0, 1) with 1 / (1 + e^-x). Tanh is its zero-centered cousin, mapping to (-1, 1). Both were the default in early neural nets, and both have a fatal flaw in deep models: they saturate. For large positive or negative inputs, the curve flattens, so its derivative goes to nearly zero.

That matters because backpropagation multiplies gradients layer by layer. When each layer's activation contributes a derivative below 1 (sigmoid's max derivative is just 0.25), the product shrinks exponentially as it flows backward. By the time the gradient reaches the early layers, it is effectively zero and those layers stop learning. This is the vanishing gradient problem, and it is why training deep sigmoid networks was so painful. Tanh is somewhat better because it is zero-centered (which keeps gradient signs balanced), but it still saturates.

ReLU and dead neurons

ReLU, max(0, x), broke the logjam. For positive inputs its derivative is exactly 1, so gradients pass through unscaled and do not vanish through depth. It is also dirt cheap to compute. This single change made very deep networks trainable and is why ReLU became the default hidden activation.

Its weakness is the flat half. For any negative input ReLU outputs zero and its gradient is zero. If a neuron's weights drift so that it always receives negative inputs, it outputs zero forever and gets no gradient to recover. It is a dead neuron, permanently stuck, and a large learning rate can kill a sizable fraction of a layer this way.

import numpy as np

def relu(x):       return np.maximum(0, x)
def leaky(x, a=0.01): return np.where(x > 0, x, a * x)
def sigmoid(x):    return 1 / (1 + np.exp(-x))
def silu(x):       return x * sigmoid(x)              # SiLU / swish

x = np.array([-3.0, -0.5, 0.0, 0.5, 3.0])
print("relu ", relu(x))    # negatives -> 0 (the dead-neuron risk)
print("leaky", leaky(x))   # negatives keep a small slope
print("silu ", np.round(silu(x), 3))  # smooth, slightly negative dip

LeakyReLU, GELU, SiLU

LeakyReLU patches the dead-neuron problem by giving the negative side a small slope (0.01x instead of 0), so a neuron always has some gradient to climb back from. PReLU makes that slope learnable.

GELU and SiLU (also called swish) are the smooth modern choices, especially in transformers. GELU weights the input by the probability it would survive under a Gaussian gate; SiLU is x * sigmoid(x). Both are smooth everywhere (no hard kink at zero) and allow a small negative output, which empirically trains a bit better in large models. They cost more to compute than ReLU, which is why ReLU still wins where speed matters most.

Softmax is for outputs

Softmax turns a vector of raw scores into a probability distribution that sums to 1, exaggerating the largest entries. It belongs on the final layer of a multiclass classifier, where it pairs with cross-entropy loss to give calibrated class probabilities. It is not a hidden-layer activation: applied between layers it would force every layer's outputs to sum to 1, which makes no sense as an internal representation.

Why interviewers probe this

The fast filter is whether you can explain why non-linearity is required at all (the linear collapse argument) without hand-waving. The deeper screen is whether you connect activation choice to a training symptom: a deep sigmoid net that will not learn points at vanishing gradients, and a layer where a chunk of neurons output constant zero points at dead ReLUs from too high a learning rate. A strong answer ends with a default: ReLU for general hidden layers, GELU or SiLU inside transformers, softmax only on the classifier output.

Common misconceptions

• "More layers always means more expressive." Not without non-linear activations between them; linear layers collapse into one.
• "ReLU has no downsides." Dead neurons. Watch for layers stuck at zero and lower the learning rate or switch to LeakyReLU/GELU.
• "Sigmoid is fine for hidden layers." In deep nets it saturates and vanishes the gradient; reserve it for binary output or gates.
• "Softmax is just another activation." It is a normalizer for output probabilities; using it on a hidden layer breaks the representation.

Key takeaways

• Activations supply non-linearity; without them a deep network is mathematically just one linear layer.
• Sigmoid and tanh saturate and cause vanishing gradients through depth; that is why ReLU replaced them in hidden layers.
• ReLU is cheap and gradient-friendly on the positive side but creates dead neurons; LeakyReLU, GELU and SiLU fix that.
• Use softmax only on the output of a multiclass classifier, paired with cross-entropy, never as a hidden activation.