Neural Networks From Scratch: Backpropagation Derived Step by Step

Every chatbot, image generator and voice assistant you have used runs on one idea: a huge pile of simple numbers, adjusted again and again by a single algorithm. That algorithm is called backpropagation. Today we build a neural network from scratch, derive every gradient by hand, and then prove in code that our math is right.

• Millions of weights
• One algorithm: backpropagation

Last episode we forecast time series with models built around trends and seasonality. Those models were designed by hand. A neural network is different, because it learns its own intermediate features directly from data, and that is why it scales to images, speech and language.

• Last episode: time series
• Today: networks that learn features

Here is the plan. First the neuron, activations and forward propagation. Then we derive backpropagation from the chain rule, layer by layer, and run it on real numbers. After that we tackle initialization, vanishing gradients, mini-batches, optimizers, dropout and batch normalization, and finish with a full NumPy network and a gradient check.

• Neuron and layers
• Backprop derivation
• Training tricks
• NumPy network

A single artificial neuron does something very simple. It takes inputs, multiplies each one by a weight, adds them up, and adds a bias. In symbols, z equals w one times x one, plus w two times x two, plus b. Large positive weights mean this input pushes toward firing, and negative weights mean it pushes against.

• Inputs times weights
• Add a bias
• Then decide

The perceptron, from the late nineteen fifties, applies a hard threshold: output one if z is at least zero, otherwise output zero. Geometrically, z equals zero is a straight line in the input plane, and the weights are the direction perpendicular to it. Points on one side are class one, points on the other side are class zero.

• Output 1 if z is at least 0
• Otherwise output 0
• Draws a straight line

The perceptron learns by correcting mistakes. If it predicts zero but the answer is one, add the input, scaled by a learning rate, to the weights. If it predicts one but the answer is zero, subtract it. For example with weights one and one, input two and negative one, and a wrong prediction that should be positive, a learning rate of zero point one moves the weights to one point two and zero point nine.

• Wrong? Nudge the weights
• w becomes w + lr times error times x

A single perceptron has a famous limit. Take the XOR problem: the points zero zero and one one are class zero, while zero one and one zero are class one. No single straight line separates them. Try any line and at least one point lands on the wrong side, so we need to stack neurons into layers.

• XOR is not linearly separable
• One line is not enough

If we stack layers that only do weighted sums, nothing is gained. Two linear layers compute W two times W one times x, and the product of two matrices is just another matrix. The whole deep stack collapses into a single linear layer. We need a nonlinear activation function between layers to bend space.

• Linear layers collapse
• W2 times W1 is just one matrix

The sigmoid squashes any number into the range zero to one: sigma of z equals one over one plus e to the minus z. Sigmoid of zero is exactly one half. Tanh is similar but centered, ranging from negative one to one. Both flatten out for large inputs, which is called saturation, and that will matter for gradients soon.

• Sigmoid squashes to 0 to 1
• Tanh squashes to -1 to 1
• Both saturate

ReLU is the workhorse of modern networks: it outputs z if z is positive and zero otherwise. It is cheap, and its slope is exactly one for positive inputs. The sigmoid derivative is sigma times one minus sigma, which never exceeds zero point two five. A common mistake is a dead ReLU that always sees negative input, so its gradient is zero forever. Leaky ReLU keeps a small slope there.

• ReLU(z) = max(0, z)
• Slope is 0 or 1
• Leaky ReLU avoids dead units

Instead of computing one neuron at a time, we compute a whole layer at once. If a layer has three inputs and five neurons, its weights form a three by five matrix, where each column belongs to one neuron. The layer computes Z equals X times W plus b, then applies the activation A equals f of Z.

• Z = XW + b
• Each column of W is one neuron

Shapes are where most bugs hide, so track them. Take a batch of four examples with three features, a hidden layer of five neurons, and one output. X is four by three, W one is three by five, so Z one is four by five. W two is five by one, so the output is four by one. Inner dimensions must match, and the outer ones survive.

• X: batch by inputs
• W: inputs by outputs
• Z: batch by outputs

Let's compute by hand. Input x is one and two. W one is the matrix with rows one, negative one, and zero point five, two. Multiplying gives one times one plus two times zero point five, which is two, and one times negative one plus two times two, which is three. Add the bias negative one, negative four, and Z is one and negative one. ReLU turns that into one and zero.

• x = [1, 2]
• Z = [1, -1]
• ReLU gives [1, 0]

Forward propagation is just this repeated layer by layer. Data enters on the left, each layer multiplies, adds bias, and activates, and a prediction exits on the right. Crucially, we cache every Z and every A along the way, because backpropagation will need them later. Forgetting to cache is a classic mistake.

• Data flows left to right
• Cache every Z and A

To learn, the network needs one number that says how wrong it is. For regression we use mean squared error. For binary classification we use cross-entropy, which is negative y times log of a, minus one minus y times log of one minus a. If the true label is one and we predict zero point five, the loss is negative natural log of one half, about zero point six nine three.

• Regression: squared error
• Classification: cross-entropy
• Single number to minimize

Recall the computational graph from our calculus episodes. Every operation is a node, every value is an edge. Forward, values flow toward the loss. Backward, each node multiplies the incoming gradient by its own local derivative. Backpropagation is nothing more than the chain rule applied to this graph in reverse order, so we never have to differentiate the whole network at once.

• Nodes are operations
• Edges carry values
• Gradients flow backward

Let's set up a neuron to differentiate. Inputs are one and two, weights are zero point five and negative zero point three, bias is zero point one, and the target is one. Then z is zero point five, minus zero point six, plus zero point one, which is exactly zero. The sigmoid gives a equals zero point five, and the loss is zero point six nine three.

• x = (1, 2), w = (0.5, -0.3)
• b = 0.1, target y = 1

Here is the core question. For every weight and bias in the network, how much does the loss change if we nudge that parameter slightly? That is the partial derivative of L with respect to the parameter, and all of them together form the gradient. Gradient descent then moves every parameter a small step in the opposite direction.

• How does L change if w changes?
• Compute dL/dw for every weight

The chain rule says that if L depends on a, a depends on z, and z depends on w, then the derivative of L with respect to w is the product of the three local derivatives. Think of gears: if one gear turns twice as fast as the next, and that one turns three times as fast as the third, the overall ratio is six. Backprop multiplies slopes along the path.

• dL/dw = dL/da times da/dz times dz/dw
• Multiply local slopes

Start at the output with sigmoid and cross-entropy. The derivative of the loss with respect to a is negative y over a, plus one minus y over one minus a. The sigmoid derivative is a times one minus a. Multiply them and the messy terms cancel, leaving dL by dz equals a minus y. The error signal is simply prediction minus truth, and we call it delta.

• dL/da = -(y/a) + (1-y)/(1-a)
• da/dz = a(1-a)
• dL/dz = a - y

Now one step back. Since z equals the previous activation times W plus b, the derivative of z with respect to W is the previous activation, and with respect to b is one. So the weight gradient is the previous activation, transposed, times delta, and the bias gradient is delta summed over the batch. The more active an input was, the more its weight is blamed.

• dL/dW = A_prev^T times delta
• dL/db = sum of delta

To continue backward we need the gradient with respect to the previous layer's activations. Each previous neuron influences every neuron it connects to, so its blame is the sum of deltas times the connecting weights. In matrix form that is delta times W transposed. This is why we need the weights saved: they route the error signal back along the same wires used going forward.

• dL/dA_prev = delta times W^T
• Each neuron collects blame from the next layer

Before reaching the weights of the earlier layer, the gradient must pass through that layer's activation. The rule is elementwise: delta for the hidden layer equals the incoming gradient times f prime of Z. For ReLU, f prime is one where Z was positive and zero elsewhere, so the gradient is simply masked. This is why we cached Z during the forward pass.

• delta = dL/dA times f'(Z)
• Elementwise multiplication

Now we can state the whole algorithm. At the output, delta equals A minus Y. For each earlier layer going backward, delta equals the next delta times the next W transposed, times f prime of Z. Then the weight gradient is the previous activation transposed times delta, and we divide by the batch size. Same three lines for a network of two layers or two hundred.

• delta_L = A_L - Y
• delta_l = (delta_(l+1) W_(l+1)^T) times f'(Z_l)
• dW_l = A_(l-1)^T delta_l

A great debugging rule: every gradient has exactly the same shape as its parameter. With our four example batch, delta two is four by one, so dW two equals A one transposed, five by four, times delta two, giving five by one. Delta one is four by five, and dW one equals X transposed, three by four, times delta one, giving three by five. If shapes disagree, you have a transposition bug.

• dW1: 3x5
• dW2: 5x1
• Gradient shape equals parameter shape

Back to our tiny neuron. The prediction a is zero point five and the target is one, so delta is negative zero point five. The weight gradient is delta times each input: negative zero point five times one, and negative zero point five times two, giving negative zero point five and negative one. The bias gradient is negative zero point five. Negative gradients mean increasing these parameters lowers the loss.

• delta = 0.5 - 1 = -0.5
• dW = (-0.5, -1.0)
• db = -0.5

Apply gradient descent with a learning rate of zero point one. The weights become zero point five five and negative zero point two, and the bias becomes zero point one five. Recompute: z equals zero point five five, minus zero point four, plus zero point one five, which is zero point three. The sigmoid gives zero point five seven four, and the loss falls from zero point six nine three to about zero point five five four.

• w = w - 0.1 times gradient
• New loss drops to 0.554

Here are the mistakes that bite most people. Forgetting to divide by the batch size makes gradients scale with batch size. Mixing up a transpose gives shape errors or, worse, silently wrong numbers. Updating weights before the whole backward pass finishes corrupts the later gradients. And applying sigmoid derivative to an already activated value twice is a surprisingly common slip.

• Forgetting to average over batch
• Wrong transpose
• Using updated weights mid-pass

Where should the weights start? Not at zero. If every weight in a layer is equal, every neuron computes the same output and receives the same gradient, so they update identically forever. The layer behaves like a single neuron. Small random numbers break this symmetry, and the bias can safely start at zero.

• Identical neurons stay identical
• Symmetry must be broken

Remember that the sigmoid slope is at most zero point two five. Backprop multiplies one such factor per layer. After ten layers the gradient is shrunk by at most zero point two five to the tenth power, about one in a million. The early layers barely learn at all. This is the vanishing gradient problem, and it is a big reason ReLU took over.

• Sigmoid slope is at most 0.25
• 0.25 to the 10th is about 0.000001

The opposite can happen too. If each layer multiplies the gradient by a factor of, say, one point five, then after twenty layers it has grown by over three thousand times. Weights then take enormous steps and the loss becomes not-a-number. Gradient clipping, which rescales the gradient when its norm exceeds a threshold, is a standard safety net, especially for recurrent networks.

• Factors above 1 compound
• Gradient clipping caps the norm

The fix is to choose the starting scale carefully so signals neither shrink nor grow as they pass through layers. Xavier initialization, designed for tanh and sigmoid, uses a variance of two over inputs plus outputs. He initialization, designed for ReLU, uses a variance of two over the number of inputs, since ReLU zeroes about half of the activations. In NumPy: random normal times square root of two over n in.

• Xavier: variance 2/(n_in + n_out)
• He: variance 2/n_in for ReLU

Computing the gradient over the entire dataset for every step is slow. Using one example is fast but noisy. Mini-batches sit in between: shuffle the data, split it into groups of, say, thirty-two or sixty-four, and update after each group. One full pass over the data is called an epoch. Matrix products also run efficiently on whole batches, which is why batches are so practical.

• Full batch: accurate but slow
• Single example: noisy
• Mini-batch: 32 to 256 typical

Plain gradient descent zigzags across narrow valleys. Momentum keeps a running velocity: v equals beta times v plus the new gradient, and the parameters move along v. Oscillating components cancel out, while the consistent downhill direction accumulates speed. Think of a heavy ball rolling down a valley instead of a feather that changes direction at every gust.

• v = beta v + gradient
• Smooths zigzags
• Builds speed in steady directions

Adam combines momentum with a per-parameter step size. It tracks a running average of the gradient, and a running average of the squared gradient, then divides one by the square root of the other. Common defaults are beta one zero point nine, beta two zero point nine nine nine, and epsilon ten to the minus eight. Parameters with big, noisy gradients get smaller steps. Even so, the learning rate remains the most important hyperparameter to tune.

• Tracks mean and squared gradient
• beta1 0.9, beta2 0.999
• Learning rate still matters

Large networks can memorize their training data. Dropout fights this by randomly switching off neurons during training, each one kept with probability p. No neuron can rely on a specific partner, so features become more robust. With inverted dropout we divide the surviving activations by p during training, so that at test time we simply use the full network with no change.

• Randomly zero neurons during training
• Keep probability p
• Scale by 1/p

Batch normalization standardizes each layer's pre-activations. For a mini-batch, subtract the mean and divide by the standard deviation, then apply learnable scale gamma and shift beta. For example, values two, four and six have mean four and standard deviation about one point six three, giving roughly negative one point two three, zero and positive one point two three. This stabilizes training and permits larger learning rates.

• Normalize each feature over the batch
• Then scale by gamma, shift by beta

Both dropout and batch normalization behave differently at training and inference time. Dropout is switched off. Batch norm stops using the current batch statistics and instead uses running averages of mean and variance collected during training. Forgetting this switch is a classic bug, where predictions on a single example look random or wildly differ from the validation score.

• Dropout off at test time
• BatchNorm uses running averages

Time to build it. Our dataset is the two moons problem: two hundred points in two interleaved crescents, which no straight line can separate. The network has two inputs, a hidden layer of sixteen ReLU units, and one sigmoid output. Weights use He initialization, biases start at zero, and we train with cross-entropy and plain gradient descent.

• Two moons, 200 points
• 2 inputs, 16 hidden ReLU, 1 sigmoid output

The code has just three functions. Forward computes Z one, A one, Z two and A two, and returns them in a cache. Backward computes delta two as A two minus Y, then dW two as A one transposed times delta two over m, then delta one with the ReLU mask, then dW one as X transposed times delta one over m. Step subtracts the learning rate times each gradient.

• forward returns cache
• backward returns grads
• step updates parameters

How do we know our backward pass is correct? Use gradient checking. Nudge one parameter up and down by a tiny epsilon, such as ten to the minus five, recompute the loss both times, and take the difference divided by two epsilon. This centered difference approximates the true derivative. Do it for each parameter and compare with the backprop value.

• Numerical slope: (L(w+e) - L(w-e)) / 2e
• Compare with backprop gradient

Compare the two gradients using relative error: the absolute difference divided by the sum of their magnitudes. As a rule of thumb, below ten to the minus seven is excellent, around ten to the minus five deserves a second look, and above ten to the minus three almost certainly means a bug. Run the check on a tiny network, with dropout off, then delete it, because it is far too slow for training.

• Relative error below 1e-7: great
• Around 1e-5: acceptable
• Above 1e-3: likely bug

Now train. The loss starts near zero point seven, which is the value for guessing one half, and falls steadily over a few thousand steps. The decision boundary starts as a nearly straight line and bends into a curve that follows the two crescents. On this small, clean toy dataset, accuracy ends up very high. Always evaluate on held-out data, though, because a training score alone can mislead.

• Loss falls over epochs
• Curved decision boundary
• Accuracy near 100% on this toy data

Let's recap. A neuron is a weighted sum plus bias, and a nonlinear activation makes stacking layers worthwhile. A layer is a matrix product, and shapes guide you. Backpropagation is the chain rule run in reverse: delta equals prediction minus truth at the output, then delta times W transpose times f prime for each earlier layer. Good initialization, mini-batches, Adam, dropout and batch norm make training work in practice.

• Neuron to layers
• Forward and backward passes
• Stable, regularized training

Writing gradients by hand teaches you what is really happening, but nobody does it for big models. In the next episode we use PyTorch, which performs backpropagation automatically, and we meet convolutional neural networks for images.

• Automatic differentiation
• Convolutional networks
• Subscribe for episode 27