Dropout and Batch Normalization Explained With Numbers and NumPy Code

Last lesson we fixed how deep networks start and how they step, with initialization and optimizers. But a network that trains well can still memorize its training data and fail on new data. Today we meet two of the most influential tricks in deep learning. One deletes neurons at random. The other rescales activations at every layer. Both look strange, and both work.

• Memorization hurts new data
• Two tricks: dropout, batch norm

We will derive dropout with real numbers, then derive batch normalization forward and backward, line by line. We will compare layer norm, weight decay in Adam, early stopping and data augmentation. Finally we implement both layers in NumPy, check their gradients numerically, and run a training experiment. We do not repeat L1 and L2 penalties, since lesson seventeen covered those.

• Dropout and batch norm derived
• Implement and gradient check
• Training curve experiment

Dropout is simple. During training, for every neuron in a layer, flip a biased coin. With probability p the neuron is dropped, meaning its output is set to zero for this one forward pass. With probability one minus p it is kept. The mask is resampled for every example and every step, so the network never sees the same wiring twice.

• Drop with probability p
• Keep with probability 1 minus p
• New mask every step

Why would deleting neurons help? Without dropout, neurons can co-adapt: one neuron fixes the mistakes of another, and together they memorize quirks of the training set. With dropout, any neuron may vanish, so each one must be useful on its own and cannot rely on a specific partner. The network is pushed toward redundant, robust features instead of fragile conspiracies.

• Neurons cannot rely on partners
• Features must stand alone

Take a layer with four activations: two, four, six and eight. Let the drop probability be one half. Suppose the random mask comes out as one, zero, one, zero. Multiplying elementwise gives two, zero, six, zero. Half the signal is gone. Notice a problem: the sum was twenty, and now it is only eight. The next layer receives a much smaller input than it will at test time.

• Activations 2, 4, 6, 8
• Mask 1, 0, 1, 0
• Sum falls from 20 to 8

The fix is inverted dropout. During training, divide the kept activations by the keep probability, one minus p. Here that means multiplying by two. Our result becomes four, zero, twelve, zero, with a sum of sixteen. For any single unit, the expected value is the keep probability times the scaled value, which equals the original activation. So on average, the signal is preserved, and no change is needed at test time.

• Divide by keep probability
• Expected activation unchanged
• No test-time rescaling

The backward pass is just as simple. Dropout is an elementwise multiplication by a fixed mask and a constant. So the gradient flowing back is multiplied by the same scaled mask. Dropped neurons receive zero gradient, and kept neurons receive their gradient times one over keep probability. You must reuse the exact mask from the forward pass, not sample a new one.

• Gradient times same mask
• Dropped units get zero gradient

There is a beautiful way to see dropout. Every mask defines a different thinned sub-network. A layer with one hundred units has two to the power of one hundred possible masks. Training with dropout is like training an enormous ensemble of sub-networks that all share the same weights, each trained a little on each step.

• Each mask is a sub-network
• Weights are shared
• Exponentially many members

At test time we would like to average the predictions of all those sub-networks, but that is impossible. Instead we use the full network with no dropout. Thanks to inverted scaling, the full network approximates the geometric mean of the ensemble, which is a good, cheap stand-in. This is an approximation, not an identity, but it works well in practice.

• Exact averaging is impossible
• Full network is the stand-in

This means a dropout layer behaves differently in two modes. In training mode it samples a mask and scales. In evaluation mode it does nothing at all and passes inputs through unchanged. A classic bug is forgetting to switch to evaluation mode, which gives noisy, random predictions, and worse validation scores than the model deserves. Always check the mode before you measure.

• Train: sample mask and scale
• Eval: identity function
• Forgetting eval is a classic bug

How much should you drop? Too little does nothing, too much destroys information and the model underfits. Common choices are around zero point one to zero point five in hidden layers, with smaller rates in layers that are already narrow. Treat p as a hyperparameter, and tune it on the validation set, as in lesson forty five.

• Too low: no effect
• Too high: underfitting
• Tune p on validation data

Now batch normalization. As weights in early layers change, the distribution of inputs to later layers keeps shifting. Each layer is chasing a moving target. Even with good initialization, activations can drift to large or tiny scales, forcing small learning rates. The idea: renormalize each layer's inputs so they stay well scaled, while still letting the network learn what it needs.

• Layer inputs drift during training
• Drift forces small learning rates

For one feature across a mini-batch of m examples, compute the batch mean, mu equals one over m times the sum of the x values. Then the batch variance, sigma squared equals one over m times the sum of x minus mu, squared. With our four numbers, two, four, six and eight, the mean is five. The squared deviations are nine, one, one and nine, so the variance is twenty divided by four, which is five.

• Mean = 5
• Squared deviations 9, 1, 1, 9
• Variance = 5

Now normalize: x hat equals x minus mu, divided by the square root of sigma squared plus epsilon. Epsilon is a tiny constant like ten to the minus five that prevents division by zero. The square root of five is about two point two three six. So our values become minus one point three four two, minus zero point four four seven, zero point four four seven, and one point three four two. They now have mean zero and variance one.

• Subtract mean, divide by std
• Epsilon avoids division by zero
• Result: 0 mean, unit variance

Forcing every layer to have zero mean and unit variance might be too restrictive. A sigmoid layer, for example, may want inputs away from zero. So batch norm adds two learnable parameters per feature: gamma to scale and beta to shift. The output y equals gamma times x hat plus beta. With gamma equal to two and beta equal to one, our values become minus one point six eight, zero point one one, one point eight nine, and three point six eight.

• Gamma scales, beta shifts
• Can undo normalization if useful
• Example: gamma 2, beta 1

Why does this help? Normalized inputs keep gradients from exploding or vanishing, allow larger learning rates, and make the network less sensitive to initialization. Batch norm also adds a bit of noise, because each example's statistics depend on its batch mates, which acts as mild regularization. The original explanation about internal covariate shift is debated, but the practical benefits are well established.

• Larger stable learning rates
• Less sensitive to initialization
• Mild noise regularizes

Now the hard part, the backward pass. Each output depends on x through three paths: directly through the numerator, through the mean, and through the variance. We receive the upstream gradient, dL by dy, and need gradients for gamma, beta and every input x. We will walk the computational graph backward, as in lesson thirty six.

• Three paths from x to output
• Need dgamma, dbeta, dx

The easy ones first. Since y equals gamma x hat plus beta, the gradient for beta is the sum of the upstream gradients over the batch. The gradient for gamma is the sum of upstream gradient times x hat. And the gradient flowing into x hat is the upstream gradient times gamma. These are the same sums you saw for the bias and weights of a linear layer.

• dbeta = sum of dy
• dgamma = sum of dy times x_hat
• dx_hat = dy times gamma

Next, the variance. Since x hat depends on sigma squared plus epsilon to the minus one half, the gradient with respect to the variance is the sum of dx hat times x minus mu, times minus one half, times that quantity to the minus three halves. For the mean, there are two contributions: a direct one, minus dx hat over the standard deviation summed, and one through the variance, which carries a factor minus two times the average of x minus mu.

• Variance: power rule, exponent minus 3/2
• Mean gets two contributions

Now sum the three paths into each x i. The gradient is dx hat i over the standard deviation, plus dL by dvar times two times x i minus mu over m, plus dL by dmu over m. After simplifying, this collapses into a compact formula: dx i equals gamma over m times standard deviation, times the quantity m times dy i, minus the sum of dy, minus x hat i times the sum of dy times x hat.

• Sum three paths into dx
• Simplifies to one expression
• Uses only sums already computed

Use our batch with gamma equal to one, and upstream gradient one, zero, zero, zero. The standard deviation is two point two three six, and x hat is minus one point three four two and so on. The sum of dy is one, and the sum of dy times x hat is minus one point three four two. Plugging in gives dx of zero point one three four, minus zero point one seven nine, minus zero point zero four five, and zero point zero eight nine. Notice they sum to zero.

• dy = 1, 0, 0, 0
• dx sums to zero
• Matches intuition

That zero sum is no accident. Normalization removes the mean, so shifting all inputs by the same constant changes nothing in the output. Therefore pushing every input up together has zero effect on the loss, and the gradients must cancel. This is a great sanity check for any batch norm implementation. A common mistake is to treat the mean and variance as constants, which gives the wrong gradient.

• Shift invariance implies zero sum
• Mean and variance are not constants

At test time we may predict a single example, so there is no batch to take statistics from. Instead, during training we keep exponential moving averages. With momentum zero point nine, the running mean updates as zero point nine times the old value plus zero point one times the batch mean. If the old value is zero and the batch mean is five, the new running mean is zero point five. After many steps it converges to the typical value.

• Moving average during training
• Used instead of batch stats
• Momentum 0.9 example

So batch norm also has two modes. In training it uses the current batch statistics and updates the running averages. In evaluation it uses the stored running statistics, making each prediction independent of the other examples. Forgetting this switch is another classic bug, and it can quietly wreck inference, especially if the running statistics are stale or the data distribution differs.

• Train: batch statistics
• Eval: running statistics
• Predictions become independent

Batch statistics are only estimates, and they get noisy when the batch is small. With a batch of two, the mean and variance are extremely unreliable, and with a batch of one the variance is zero. Training statistics then mismatch the running statistics used at test time. This is why batch norm struggles with tiny batches, sequences of varying length, and some distributed setups.

• Noisy mean and variance
• Batch of one has no variance
• Train and test mismatch

Layer normalization avoids the problem by normalizing across the features of a single example instead of across the batch. Same formula, different axis. There are no running statistics, and training and inference behave identically. That makes it the standard choice in transformers, which we will build properly in lesson fifty six. Batch norm remains popular in convolutional networks with decent batch sizes.

• Normalize per example
• No running statistics
• Used in transformers, lesson 56

In lesson seventeen, L2 regularization added a penalty to the loss. With plain gradient descent, that is identical to weight decay, which shrinks weights by a small factor each step. With Adam, they differ. L2 adds lambda times w to the gradient, which then gets divided by the adaptive square root of the second moment estimate. Weights with large gradient history are decayed less, which is not what we wanted.

• Same for plain gradient descent
• Adam rescales L2 gradient
• Decay becomes uneven

AdamW fixes this by decoupling the decay. It performs the Adam step using only the loss gradient, and then separately shrinks the weights by learning rate times lambda times w. Every weight is decayed in proportion to its size, no matter its gradient history. In practice AdamW is the default for training transformers, and it makes the decay setting easier to tune independently of the learning rate.

• Adam step from loss gradient only
• Separate shrink toward zero
• Default for transformers

Early stopping is regularization by timing. Watch the validation loss each epoch and save the model whenever it improves. If it fails to improve for a set number of epochs, called the patience, stop and restore the best saved weights. For example, with patience five, if the best loss was at epoch twenty, you stop at epoch twenty five and return to epoch twenty's weights. Never use the test set for this decision.

• Monitor validation loss
• Patience of five epochs
• Restore the best weights

The most reliable cure for overfitting is more data, and augmentation manufactures it. For images, flip, crop, rotate and adjust brightness, so the label stays the same while the pixels change. For text or audio there are analogous tricks. The key rule is that transformations must preserve the label: flipping a photo of a cat is fine, but flipping the digit six may make it look like something else.

• Label-preserving transformations
• More variety, same labels
• Beware changing the label

Time to implement. The dropout layer stores a drop probability and a mask. In the forward pass, when training, it draws random numbers, keeps those that are at least p, and divides by one minus p, then multiplies by the input. In evaluation it returns the input unchanged. The backward pass simply multiplies the upstream gradient by the stored mask. About ten lines of code.

• Mask scaled by 1 over keep probability
• Eval returns input
• Backward reuses the mask

The batch norm layer stores gamma, beta, running mean and running variance. Forward in training computes the batch mean and variance, normalizes, scales and shifts, caches what backward needs, and updates the running averages. In evaluation, it uses the running values. Backward computes dgamma and dbeta as sums, then dx with the compact formula from earlier, using the cached x hat and standard deviation.

• Cache x_hat and std
• Update running statistics
• Backward uses compact formula

How do we know the backward pass is right? Use a numerical gradient check. Nudge one input by a small amount, like ten to the minus five, in each direction, compute the loss both times, and divide the difference by two times that amount. Compare with your analytic gradient using relative error. Values around ten to the minus seven or smaller are a pass. For dropout, fix the random mask during the check, or the loss changes between evaluations.

• Centered difference with small h
• Relative error near 1e-7
• Freeze dropout mask while checking

Now the experiment. Train a small multilayer network on a modest dataset four ways: baseline, with dropout, with batch norm, and with both. Plot training and validation loss against epochs. You would typically expect the baseline to overfit, with validation loss rising late. Dropout should shrink the gap, at the cost of slower training loss. Batch norm should make both curves drop faster, and allow a larger learning rate.

• Baseline, dropout, batch norm, both
• Watch the gap and the speed

Read the plots carefully. Do not claim an effect from one run, because random seeds change the curves, so repeat with several seeds. Dropout and batch norm interact: batch norm often reduces the need for dropout, and putting dropout before a batch norm layer can shift the variance statistics between train and eval. If results look odd, check the modes first, and then the learning rate.

• Repeat across seeds
• Layers interact
• Check modes first

Let us collect the classic mistakes. Forgetting to switch between training and evaluation modes. Sampling a new dropout mask in the backward pass. Ignoring the scaling by one over keep probability. Treating the batch mean and variance as constants in backward. Using batch norm with tiny batches. And using L2 inside Adam when you really wanted decoupled weight decay.

• Wrong mode at inference
• Wrong mask or scaling
• Tiny batches, wrong decay

To recap. Dropout zeroes random units and rescales the survivors, so training approximates an ensemble of thinned networks, and evaluation uses the full network. Batch norm normalizes by batch mean and variance, then applies learnable gamma and beta, using running statistics at inference. Layer norm swaps the axis. AdamW, early stopping and augmentation round out your toolkit.

• Dropout: mask plus 1/(1-p)
• BatchNorm: normalize, gamma, beta
• Plus AdamW, early stopping, augmentation

You now understand two layers that shaped modern deep learning, and you built them yourself. Next time, in lesson forty nine, we move into PyTorch tensors and autograd, where this backward pass is computed for you.

• Lesson 49: tensors and autograd
• Subscribe and follow the playlist