Convolution Explained: Kernels, Stride, Padding and Pooling by Hand

Take one ordinary photo and connect it to just one thousand neurons in a dense layer. You need about one hundred fifty million weights. Now do the same job with a convolution and you can get by with around eighteen hundred. This lesson shows exactly how that trick works, number by number.

• Dense layer on one photo
• Convolution: tiny shared kernel

In lesson 50 we built training loops in PyTorch, moved them to the GPU, and learned to debug them. Today we open the box on the layer that made vision work, the convolution. We will do it by hand, derive the size formulas, count parameters, and then write it from scratch in NumPy and check it against PyTorch.

• Lesson 50: loops, GPUs, debugging
• Today: convolution mechanics

A grayscale image is just a grid of numbers, each between zero and two hundred fifty-five. A colour image stacks three such grids, for red, green and blue. So we describe it as a tensor with shape height, width, channels. A 224 by 224 colour photo has shape 224, 224, 3.

• Shape (H, W, C)
• Pixels 0 to 255
• RGB has C = 3

To feed that photo to a dense layer, we flatten it into a vector of 224 times 224 times 3, which is 150,528 numbers. With just one thousand hidden neurons, every neuron needs a weight for every input. That is 150,528,000 weights, plus a thousand biases. One layer, one small photo, and already over a hundred and fifty million parameters.

• 224 x 224 x 3 = 150,528 inputs
• 1,000 neurons
• About 150.5 million weights

The cost is only part of the problem. Flattening throws away the fact that neighbouring pixels belong together. And a cat in the top left uses completely different weights than a cat in the bottom right, so the network must relearn the same pattern at every position. We want a layer that looks locally and reuses what it learns everywhere.

• Flattening destroys neighbourhoods
• No translation sharing

Start in one dimension. Our signal is one, two, three, four, five, and our kernel is one, zero, minus one. Slide the kernel along, multiply overlapping entries, and add. At the first position we get one times one, plus two times zero, plus three times minus one, which is minus two. Every position gives minus two, so the output is minus two, minus two, minus two.

• Signal [1, 2, 3, 4, 5]
• Kernel [1, 0, -1]
• Slide, multiply, sum

Strictly, mathematical convolution flips the kernel before sliding it. Without the flip, the operation is called cross-correlation. Flip our kernel to minus one, zero, one, and every output becomes plus two instead of minus two. Deep learning libraries skip the flip and still call it convolution, because the kernel is learned anyway, so the network just learns the flipped version.

• True convolution flips the kernel
• Deep learning skips the flip
• Library 'conv' is cross-correlation

Now two dimensions. Our five by five image has two dark columns of zeros on the left and three bright columns of nines on the right, so there is a vertical edge between column one and column two. The kernel is three by three with minus one in the left column, zeros in the middle, and plus one in the right column. It responds when the right side is brighter than the left.

• Dark left, bright right
• Kernel: columns -1, 0, 1
• Detects vertical edges

Place the kernel on the top left patch, columns zero to two. In each row we compute zero times minus one, zero times zero, plus nine times one, which is nine. There are three rows, so the total is twenty-seven. That is our first output value, and it is large because the patch contains the edge.

• Top-left 3x3 patch
• Three rows: 0 + 0 + 9 = 9
• Total = 27

Slide one column right. Each row now gives zero times minus one, nine times zero, nine times one, again nine, so the total is twenty-seven. Slide once more and each row is minus nine, plus zero, plus nine, which is zero, because the patch is uniformly bright. Repeating for all three rows, the output is a three by three map where every row reads twenty-seven, twenty-seven, zero.

• Shift right: 27
• Shift again: 0
• Output is 3x3: every row 27, 27, 0

Notice what happened. The same nine weights were used at every position, and each output looked only at a small neighbourhood. If the edge moved one pixel, the response would move one pixel too. This property is called translation equivariance, and it is why a handful of shared weights can replace millions.

• Same 9 weights at every position
• Local connections
• Translation equivariance

Why was our output only three by three? A three wide kernel fits into a five wide image at three positions, which is five minus three plus one. Every unpadded convolution shrinks the image by k minus one. Stack many layers and the picture quickly vanishes, and pixels at the border get used far less than those at the centre.

• n = 5, k = 3
• Positions: 5 - 3 + 1 = 3
• Image shrinks each layer

Padding fixes this by adding a frame of zeros around the image. With a three by three kernel, padding of one keeps a 32 by 32 image at 32 by 32, so we call it same padding. In general, for odd kernels and stride one, the same padding is k minus one, divided by two. Zero padding is the default, and reflecting or replicating border values are alternatives.

• Add p zeros around the border
• 'Same' padding: p = (k-1)/2
• k = 3 means p = 1

Stride is how far the kernel jumps between positions. With stride one it visits every location. With stride two it skips every other one, so the output has roughly half the height and width, and a quarter of the values. This is one way networks reduce resolution while going deeper.

• Jump s pixels per step
• Stride 2 halves resolution
• Cheaper, coarser output

All of this combines into one formula. The output size is the floor of n plus two p minus k, divided by s, then plus one. Here n is the input size, p the padding on each side, k the kernel size and s the stride. The floor throws away a final partial step that does not fit.

• floor((n + 2p - k) / s) + 1
• n input, k kernel, p padding, s stride
• Floor drops partial steps

Check it on three cases. First, n of 32, kernel three, padding one, stride one: 32 plus 2 minus 3 is 31, divided by one is 31, plus one gives 32. Second, n of 7, kernel 3, no padding, stride 2: 4 divided by 2 is 2, plus one gives 3. Third, n of 224, kernel 7, padding 3, stride 2: 223 divided by 2 is 111.5, floor to 111, plus one gives 112.

• 32, k=3, p=1, s=1 gives 32
• 7, k=3, p=0, s=2 gives 3
• 224, k=7, p=3, s=2 gives 112

Dilation spreads the kernel taps apart by inserting gaps. A three wide kernel with dilation two covers five pixels, because the effective size is d times k minus one, plus one. It sees a wider area with the same three weights per row. Just swap k for the effective size in the output formula. For n of 11, dilation 2 gives effective size 5, and the output is 7.

• Gaps between kernel taps
• Effective size: d(k-1) + 1
• Bigger view, same 9 weights

Real inputs have channels, so a kernel must too. A filter for an RGB image is three by three by three. At each position we multiply all twenty-seven entries with the patch beneath, add them up, and get one single number. So one filter produces one output channel, called a feature map.

• Kernel shape k x k x C_in
• Sum over all channels
• One filter gives one map

A layer uses many filters, each learning to detect a different pattern. Sixty-four filters yield sixty-four feature maps, stacked into the next tensor. The parameter count is k times k times input channels times output channels, plus one bias per filter. This is where the learning lives.

• C_out filters
• Stack maps into new tensor
• Params: k*k*C_in*C_out + C_out

First layer: three by three kernel, three input channels, sixty-four filters. That is 3 times 3 times 3 times 64, which is 1,728 weights, plus 64 biases, giving 1,792. Another layer with 16 inputs and 32 outputs has 3 times 3 times 16 times 32, which is 4,608, plus 32 biases, equal to 4,640. Notice the count does not depend on image height or width at all, which is the answer to our hook.

• 3x3, 3 in, 64 out: 1,792
• 3x3, 16 in, 32 out: 4,640
• Independent of image size

A one by one convolution looks at a single pixel but all its channels. It is a small dense layer applied independently at every location. It changes the number of channels without touching the spatial size. Going from 256 channels to 64 costs 256 times 64 plus 64 biases, which is 16,448 parameters, and it is a cheap way to compress or mix features.

• Mixes channels, not space
• Dense layer at every pixel
• 256 to 64: 16,448 params

The receptive field of a unit is the region of the original image that can influence it. One three by three layer sees three pixels across. Stack a second and each unit sees a three by three block of units that each saw three, so five across. A third gives seven. Depth lets small kernels build a large view.

• Input region affecting one output
• One 3x3 layer: 3
• Two: 5. Three: 7

Strides make the field grow faster. Each layer adds k minus one times the current jump, the pixel spacing between neighbouring units, and the jump multiplies by that layer's stride. Take a three by three conv, a two by two pool with stride two, then another three by three conv. The field goes to 3, then 4, then 4 plus 2 times 2, which is 8.

• r_out = r_in + (k-1) * jump
• jump multiplies by stride
• Conv3, pool2, conv3: r = 8

Three stacked three by three layers see the same seven by seven region as one big seven by seven kernel. But with C channels in and out, they cost 27 C squared weights versus 49 C squared. With 64 channels, that is 110,592 against 200,704. We also get three nonlinearities instead of one, so deeper stacks are both cheaper and more expressive.

• Three 3x3 equal one 7x7 view
• 27C^2 vs 49C^2 weights
• C = 64: 110,592 vs 200,704

Pooling shrinks feature maps with no learned weights. Take a four by four map with rows one, three, two, one; four, six, five, zero; seven, two, nine, eight; one, zero, three, four. Max pooling with a two by two window and stride two keeps the largest value in each block: six, five, seven, nine. The output is two by two.

• 2x2 window, stride 2
• Keep the largest value
• Output [6, 5; 7, 9]

Average pooling takes the mean of each block instead. On the same map we get 3.5, 2, 2.5 and 6. Global average pooling goes all the way, averaging each whole channel into one number, so a seven by seven by 512 tensor becomes a vector of 512. That replaces a huge flatten and dense step, and it works for any input size.

• Average: 3.5, 2, 2.5, 6
• Global average: one number per channel
• 7x7x512 becomes 512

Pooling gives a little tolerance to small shifts, since a feature moving one pixel inside a block gives the same maximum. It also cuts the cost of later layers by a factor of four for each two by two pool. The price is lost spatial detail. Modern designs sometimes replace pooling with strided convolutions, which learn how to downsample.

• Small shifts barely change output
• Less compute downstream
• Info is discarded

Backpropagation through a convolution reuses the same operation. The gradient for the kernel is the cross-correlation of the input with the upstream gradient, because each weight touched every position, so contributions add up. The gradient for the input is a full convolution of the upstream gradient with the flipped kernel. The bias gradient is just the sum of the upstream gradient over positions.

• Bias gradient: sum of upstream
• Kernel gradient: correlate input with upstream
• Input gradient: full conv with flipped kernel

Pooling has no weights, but gradients still flow through it. For max pooling, the whole gradient goes to the position that won the max, and the others get zero. For average pooling, the gradient is split equally, so each of four entries gets a quarter. Autograd in PyTorch handles all of this for you, as we saw in lesson 49.

• Max pool: gradient to the argmax
• Average pool: split evenly
• Shared weights accumulate

Now let's code it. Our function takes an input of shape batch, input channels, height, width, weights of shape output channels, input channels, k, k, a bias, a stride and a padding. First we pad with zeros using NumPy. Then we compute the output size with our formula, and allocate an array of that shape filled with zeros.

• Input (N, C_in, H, W)
• Weights (C_out, C_in, k, k)
• Loops over output position

The core is a set of nested loops over each example, each filter, and each output row and column. For each one, we slice out the patch of the padded input, across all input channels, multiply it elementwise with that filter, sum, and add the bias. The patch start is the output index times the stride. It is slow, but every line maps directly to what we did by hand.

• Loop over batch, filters, rows, columns
• Slice the patch
• Sum of patch * kernel + bias

First test: feed in our five by five image and edge kernel with no padding. We should get the three by three map with rows twenty-seven, twenty-seven, zero, and we do. Then compare against PyTorch's functional conv2d on random tensors, with several strides and paddings, and check the maximum absolute difference. It should be around one in a million, just floating point rounding, so use allclose, not exact equality.

• Expect rows 27, 27, 0
• Compare with torch.nn.functional.conv2d
• Differences near 1e-6

Watch for three classic bugs. First, flipping the kernel when your library does not. Second, mixing up channel order: PyTorch expects channels first, while many image libraries give channels last. Third, off-by-one errors in output size, so always compare your shapes against the formula before comparing values. Most of these show up as a shape error long before any wrong number.

• Forgetting the flip is not needed
• Mixing up (H,W,C) and (C,H,W)
• Off-by-one in output size

Let's recap. Images are tensors, and dense layers on them explode in parameters, while a convolution slides a small shared kernel and gets translation equivariance. The output size is the floor of n plus two p minus k over s, plus one, and the parameter count is k squared times input channels times output channels plus biases. One by one convolutions mix channels, depth grows the receptive field, pooling downsamples, and backprop reuses convolution itself.

• Local, shared weights beat dense
• Output: floor((n+2p-k)/s)+1
• Params: k*k*C_in*C_out + C_out
• Pooling downsamples; gradients reuse conv

You now understand every moving part inside a convolutional layer. In lesson 52 we assemble these parts into full CNN architectures, learn how transfer learning reuses pretrained networks, and visualize what the filters actually detect.

• Lesson 52: architectures
• Transfer learning
• Visualizing features