Neural Network Basics: Perceptron to MLP

TL;DR: A perceptron is one linear unit plus a threshold, so it can only learn problems a single straight line can separate. XOR is the canonical counterexample it cannot solve. Add a hidden layer and a non-linear activation and you get a multilayer perceptron, which can approximate essentially any function because the hidden units bend the input space. The forward pass computes the prediction layer by layer; backpropagation is the chain rule applied to push gradients back so gradient descent can update every weight.

The perceptron

The perceptron is the atom of a neural network. It takes inputs x1...xn, multiplies each by a weight, sums them with a bias, and passes the result through a step function: output 1 if the sum clears a threshold, else 0. Geometrically it draws a single hyperplane and labels everything on one side 1 and the other 0. That is powerful enough for linearly separable problems like AND and OR, and a single perceptron can be trained with a simple weight-update rule.

import numpy as np

def perceptron(x, w, b):
    return 1 if np.dot(w, x) + b > 0 else 0

# Learns OR with a hand-picked line
w, b = np.array([1.0, 1.0]), -0.5
for x in [(0,0),(0,1),(1,0),(1,1)]:
    print(x, perceptron(np.array(x), w, b))

Why one layer cannot do XOR

XOR outputs 1 when exactly one input is on: (0,1) and (1,0) are positive, (0,0) and (1,1) are negative. Plot those four points and the two positive ones sit on opposite diagonal corners. No single straight line can put both positives on one side and both negatives on the other. This is the Minsky and Papert result from 1969 that stalled neural-net research for years: a single-layer perceptron is fundamentally limited to linearly separable functions.

The fix is not a better training rule. It is more layers. Two hidden units can each carve one of the needed boundaries, and a third unit combines them. The combination is no longer linear in the original inputs, so the impossible becomes routine.

Hidden layers and non-linear activations

A multilayer perceptron stacks layers: input, one or more hidden layers, output. Each hidden unit computes a weighted sum then applies a non-linear activation (ReLU, sigmoid, tanh). The non-linearity is the whole point. If every activation were linear, stacking layers would collapse to a single linear map, no more expressive than one perceptron. The non-linear bends are what let the network warp the input space until the classes become separable.

The universal approximation theorem makes this precise: an MLP with one sufficiently wide hidden layer and a non-linear activation can approximate any continuous function on a bounded domain to arbitrary accuracy. In practice you go deeper rather than absurdly wide, because depth composes features more efficiently. The hidden layers learn their own features, which is the leap over classical models where you hand-engineer them.

The spine above follows one row the whole way, forward to a prediction and backward to a gradient, and marks the activation as the step every later layer depends on.

The forward pass and where backprop fits

The forward pass runs left to right: multiply by weights, add bias, apply activation, feed to the next layer, until the output produces a prediction and the loss measures how wrong it is.

Backpropagation is not a separate learning algorithm. It is an efficient application of the chain rule that computes the gradient of the loss with respect to every weight in one backward sweep, reusing intermediate results so the backward pass costs about as much as two forward passes, rather than one pass per weight. Those gradients then feed gradient descent, which does the actual updating. People conflate the two; the distinction matters in interviews.

Why interviewers probe this

The XOR question separates people who memorized "neural nets are universal approximators" from people who understand why depth and non-linearity are required. The tell is whether you can say, without prompting, that a linear activation makes stacking pointless. The held-back follow-up is usually "so what does backprop actually compute," and the strong answer names the chain rule and that it returns a gradient, with gradient descent doing the update.

Common misconceptions

• "More layers always help." Past a point you add variance and training instability without reducing bias; depth needs normalization and good initialization to train.
• "Backprop is the optimizer." Backprop computes gradients; the optimizer (SGD, Adam) uses them.
• "The activation is a minor detail." Remove the non-linearity and a deep net collapses to a single linear layer.
• "A perceptron failing XOR means neural nets cannot do XOR." A single layer cannot; two layers do it trivially.

Key takeaways

• A perceptron draws one hyperplane, so it only solves linearly separable problems and cannot learn XOR.
• Hidden layers plus a non-linear activation make an MLP a universal approximator that learns its own features.
• Linear activations collapse depth to a single layer; the non-linearity is mandatory.
• The forward pass predicts; backprop is the chain rule computing gradients; gradient descent does the update.