PyTorch and Convolutional Neural Networks: Tensors to Transfer Learning

A photo is just a grid of numbers. Somehow, a few dozen lines of code can turn that grid into the word cat, with accuracy that once seemed impossible. Today we learn how, using PyTorch to build the machinery and convolutional neural networks to give it eyes.

• Pixels in, label out
• PyTorch + CNNs

Last episode we built a neural network from scratch in NumPy and derived backpropagation by hand. It worked, but every new layer meant writing new gradient code. This episode removes that burden, and then teaches the network to exploit the structure of images.

• Last time: network from scratch
• Manual backpropagation

Here is the plan. First, PyTorch: tensors, autograd, datasets, the training loop, GPUs, saving and debugging. Second, convolution, which we will compute by hand with real numbers. Third, we build and train a CNN, then improve it with transfer learning, augmentation, and a look at what the filters learned.

• PyTorch essentials
• Convolution by hand
• Build, train, transfer

A tensor is an n-dimensional array of numbers, just like a NumPy array. It has a shape, a data type, and a device that says where it lives, on the CPU or on a GPU. A scalar has zero dimensions, a vector has one, a matrix has two, and a batch of color images has four.

• N-dimensional array
• Shape, dtype, device

You create tensors with torch dot tensor, torch dot zeros, or torch dot randn for random normal values. Reshaping changes the shape without changing the data, so a tensor with twelve elements can become three by four or two by six. The most common bug in deep learning is a shape mismatch, so get in the habit of printing shapes constantly.

• torch.tensor, zeros, randn
• view and reshape

Operations on tensors apply to every element at once, which is far faster than Python loops. Broadcasting lets shapes that differ be combined: adding a vector of three biases to a matrix of shape four by three adds that vector to every row. Matrix multiplication uses the at symbol, and a linear layer is just that product plus a bias.

• Operations are vectorized
• Broadcasting stretches shapes

Autograd is the heart of PyTorch. When a tensor has requires grad set to true, every operation on it is recorded in a computational graph. Calling backward on a final scalar walks that graph in reverse, applying the chain rule exactly as we did by hand, and stores the results in each tensor's grad attribute.

• requires_grad=True
• Computational graph recorded

Let w equal three, x equal two, b equal one, and the target y equal ten. The prediction is three times two plus one, which is seven, so the error is minus three and the loss is nine. The gradient of the loss with respect to w is two times the error times x, which is two times minus three times two, giving minus twelve. PyTorch reports exactly that after backward.

• w=3, x=2, b=1, y=10
• Loss = (wx+b-y)^2

Here is a classic mistake. Gradients accumulate: each call to backward adds to the existing grad instead of replacing it. If you forget to zero the gradients every step, your updates mix together old and new information and training behaves strangely. For evaluation, wrap code in torch dot no grad so PyTorch does not waste memory recording a graph.

• grad adds up each backward()
• Call zero_grad() every step
• torch.no_grad() for inference

Writing raw tensors gets tedious, so PyTorch gives us nn dot Module. You subclass it, create layers such as nn dot Linear in the constructor, and describe how data flows in the forward method. Every layer's weights are registered automatically, so model dot parameters hands the optimizer everything it needs to update.

• nn.Module holds parameters
• forward defines computation

Data enters PyTorch through two classes. A Dataset knows how to return one sample, through a length method and a get item method. A DataLoader wraps a dataset and serves shuffled mini-batches, optionally using several worker processes. With sixty thousand images and a batch size of sixty-four, one epoch is about nine hundred and thirty-eight batches.

• Dataset: one sample at a time
• DataLoader: batches, shuffling

Every PyTorch training loop is the same five beats. Run the forward pass to get predictions. Compute the loss. Zero the gradients. Call backward on the loss. Then call optimizer dot step to update the weights. Wrap that in a loop over batches, and wrap that in a loop over epochs, and you are training.

• Forward, loss, zero_grad
• Backward, step

For classification you use cross entropy loss, which expects raw scores called logits, not probabilities. It applies softmax internally, so adding your own softmax layer before it is a common mistake. For the optimizer, stochastic gradient descent with momentum is a solid classic, and Adam, with a learning rate around one in a thousand, is a convenient default that often works well immediately.

• CrossEntropyLoss takes logits
• SGD, Adam

Some layers behave differently during training and evaluation. Dropout randomly switches off units while training, and batch normalization uses batch statistics in training but running averages at test time. So call model dot train before training and model dot eval before validating. Forgetting eval is a subtle bug that makes validation numbers noisy and misleading.

• model.train() vs model.eval()
• Validate every epoch

Always record training loss and validation loss each epoch. If both fall together, learning is healthy. If training loss keeps falling while validation loss turns upward, the model is memorizing, exactly the overfitting we studied in episode thirteen. That gap tells you when to stop, add regularization, or get more data.

• Train loss falling, val loss rising = overfit
• Track both

A GPU runs thousands of simple operations in parallel, which is perfect for the matrix math in neural networks. In PyTorch you pick a device, then move both the model and every batch to it with dot to device. If one is on the GPU and the other on the CPU, you get an error complaining about mismatched devices, and the fix is almost always a missing dot to.

• device = cuda if available
• Model and data on same device

Mixed precision is a speed trick. Instead of using thirty-two bit floats everywhere, PyTorch can run many operations in sixteen-bit formats, which saves memory and often speeds up training on modern GPUs. It does this inside an autocast context, and keeps sensitive pieces in higher precision. We will only mention it here, but remember the numerical stability lessons from earlier episodes explain why that care is needed.

• 16-bit math where safe
• Faster, less memory

To save a model, the recommended approach is to save its state dict, which is a dictionary mapping each layer name to its tensors. To load, rebuild the same architecture, call load state dict on it, and then switch to eval mode. You can also save the optimizer state and epoch number in a checkpoint, so an interrupted run can resume instead of starting over.

• Save state_dict
• Load, then model.eval()

The best debugging trick is to overfit a single small batch. If your model cannot drive the loss on sixteen examples close to zero, something is broken: the labels, the loss, the learning rate, or a missing zero grad. Also check the starting loss. For ten balanced classes with cross entropy it should be near the natural log of ten, about two point three.

• Overfit one batch first
• Print shapes, check loss

Training involves randomness: weight initialization, shuffling, dropout, augmentation. Set seeds for Python, NumPy and PyTorch so a run can be repeated. Be aware that some GPU operations are nondeterministic, so even with seeds, tiny differences can appear. For honest comparisons, run several seeds and report the average and the spread, not one lucky run.

• Set seeds
• Some GPU ops nondeterministic

A grayscale image is a matrix. Each entry is a pixel brightness, from zero for black to two hundred and fifty-five for white. A twenty-eight by twenty-eight digit image therefore has seven hundred and eighty-four numbers. Typically we divide by two hundred and fifty-five so values lie between zero and one.

• Grayscale: height x width
• Pixel values 0 to 255

A color image has three channels: red, green and blue. PyTorch stores a batch as a four-dimensional tensor with shape batch, channels, height, width. So thirty-two color images of size two hundred and twenty-four by two hundred and twenty-four have shape thirty-two, three, two hundred and twenty-four, two hundred and twenty-four. Mixing up channel order is another frequent source of bugs.

• Channels: red, green, blue
• PyTorch shape: N, C, H, W

Why not flatten the image and use a normal network? A twenty-eight by twenty-eight image becomes seven hundred and eighty-four inputs, and one hidden layer of a hundred units needs seventy-eight thousand four hundred weights plus a hundred biases. Worse, flattening throws away neighborhood structure, and a shifted digit looks like a totally different input. We want weights that are few, local, and reused across the image.

• 784 inputs x 100 units = 78,400 weights
• Loses spatial structure

The convolution idea is simple. Take a small grid of weights called a kernel, slide it across the image, and at each position multiply overlapping numbers and add them up. The same weights are used at every location, so a pattern detector learned in one corner works everywhere. This is called weight sharing, and it is why CNNs are so efficient.

• Small kernel slides over image
• Same weights everywhere

A small honest note on naming. In textbook math, convolution flips the kernel before sliding. Deep learning libraries skip the flip and compute what is technically cross-correlation, but still call it convolution. Since the kernel weights are learned anyway, the flip makes no practical difference, so we will compute it the way PyTorch does.

• DL convolution = cross-correlation
• No kernel flip

Let us compute one by hand. Our input is a four by four grid with rows: one, two, zero, one; three, one, two, zero; zero, one, one, two; and two, zero, one, three. Our kernel is three by three, with every row equal to one, zero, minus one. This kernel compares the left side of a patch against the right side.

• 4x4 input
• 3x3 vertical-edge kernel

Place the kernel on the top left. The patch is one, two, zero; three, one, two; zero, one, one. Row one gives one minus zero, which is one. Row two gives three minus two, which is one. Row three gives zero minus one, which is minus one. Adding one, one and minus one gives one, so the first output is one.

• Patch: 1 2 0 / 3 1 2 / 0 1 1
• Result: 1

Slide one step right: the patch is two, zero, one; one, two, zero; one, one, two, giving one, one and minus one, total one. Slide back to the left and down one row: the total is again one. The last patch, one, two, zero; one, one, two; zero, one, three, gives one, minus one and minus three, total minus three. Our output is one, one, one, minus three.

• Output is 2x2
• [[1, 1], [1, -3]]

Different kernels detect different things. The kernel we used responds strongly where brightness changes from left to right, a vertical edge. An averaging kernel blurs, and other patterns sharpen or find horizontal edges. In classic computer vision, people designed these by hand. In a CNN, the kernel values are parameters and gradient descent learns them.

• Edges, blurs, sharpening
• Kernels are features

Stride is how many pixels the window moves each step. With stride one it moves one pixel at a time. With stride two it skips every other position, so the output is roughly half the size in each direction. Larger strides shrink feature maps quickly and cut computation, at the cost of some detail.

• Step size of the window
• Stride 2 halves the output

Without padding, each convolution shrinks the image and edge pixels get used less often. Padding adds a border of zeros around the input. With a three by three kernel, padding of one keeps the height and width unchanged, which we call same padding. Padding of zero is called valid padding.

• Add a border of zeros
• Padding 1 keeps 3x3 size

Here is the formula. Output size equals the floor of n plus two p minus k, all divided by s, plus one. Here n is the input size, p is padding on each side, k is the kernel size, and s is the stride. The intuition: the padded input has n plus two p pixels, the window needs k of them to start, and then each extra stride moves it forward.

• out = floor((n + 2p - k) / s) + 1
• n input, k kernel, p padding, s stride

Try it. A thirty-two pixel input, kernel five, padding zero, stride one: thirty-two minus five is twenty-seven, divided by one, plus one, gives twenty-eight. Now a two hundred and twenty-four input, kernel seven, padding three, stride two: two hundred and twenty-four plus six minus seven is two hundred and twenty-three, divided by two is one hundred and eleven point five, floor gives one hundred and eleven, plus one is one hundred and twelve. Remember that floor step; forgetting it is a common mistake.

• 32, k5, p0, s1 gives 28
• 224, k7, p3, s2 gives 112

What happens with color images? The kernel gains a depth to match the input channels. For an RGB image, a three by three kernel is really three by three by three, twenty-seven weights. You multiply each channel with its own slice, then add everything, including across channels, into a single number per position.

• Kernel is k x k x C_in
• Sum across channels

A layer uses many kernels, each producing its own feature map. Sixteen filters give sixteen output channels. Count parameters for three input channels, sixteen filters, kernel three: each filter has twenty-seven weights plus one bias, so twenty-eight, times sixteen is four hundred and forty-eight. Notice this does not depend on image size at all.

• Each filter makes a feature map
• Params = C_out x (k x k x C_in + 1)

After convolution we apply a ReLU, which keeps positive values and zeros the rest, adding the non-linearity we need. Then pooling shrinks the map. Max pooling with a two by two window and stride two takes the largest value in each block. Pooling reduces computation and makes the network tolerant to small shifts of a feature.

• ReLU after convolution
• Max pool 2x2 halves size

Take a four by four map with rows one, three, two, zero; five, two, one, one; zero, four, six, two; and one, one, three, and five. The top left block has maximum five, the top right has two, the bottom left has four, and the bottom right has six. Output: five, two, four, six. Pooling has no learnable parameters, and it keeps the strongest response while discarding exact position.

• Blocks: max of each 2x2
• Pooling has no parameters

The receptive field of a unit is the region of the original image that can influence it. One three by three convolution sees three by three pixels. Stack a second one and each unit now sees five by five; a third makes seven by seven. Pooling and stride make fields grow even faster, so deeper layers see larger context.

• Region of input a unit sees
• Two 3x3 layers see 5x5

Three stacked three by three layers see the same seven by seven region as one seven by seven kernel. With C channels in and out, that is three times nine times C squared, twenty-seven C squared weights, versus forty-nine C squared. So stacking is cheaper and adds more non-linearities. Early layers detect edges, middle layers combine them into textures, and later layers assemble parts and objects.

• Edges, then textures, then parts
• Three 3x3 beat one 7x7

Now we build a CNN for twenty-eight by twenty-eight grayscale digits. Block one: convolve one channel into sixteen with kernel three and padding one, apply ReLU, and pool. The shape goes from one by twenty-eight by twenty-eight to sixteen by fourteen by fourteen. Block two: sixteen into thirty-two channels, then pool again, giving thirty-two by seven by seven.

• Conv-ReLU-Pool, twice
• Flatten, then Linear

Flatten thirty-two times seven times seven to get fifteen hundred and sixty-eight features, then a linear layer maps them to ten class scores. Count the parameters. First convolution: sixteen times nine plus sixteen, one hundred and sixty. Second: thirty-two times sixteen times nine plus thirty-two, four thousand six hundred and forty. Linear: fifteen hundred and sixty-eight times ten plus ten, fifteen thousand six hundred and ninety. Total, twenty thousand four hundred and ninety, far fewer than the flattened network.

• Conv1: 160
• Conv2: 4,640
• Linear: 15,690
• Total: 20,490

In code this is short. Use nn dot Sequential with Conv2d, ReLU, MaxPool2d, repeated twice, then Flatten and Linear. Conv2d takes input channels, output channels, kernel size and padding. A great habit is to pass a fake batch through the model and print the shape after each layer to confirm the numbers we just computed.

• nn.Conv2d, nn.MaxPool2d
• nn.Flatten, nn.Linear

We train with cross entropy loss and Adam at a learning rate of one in a thousand, using batches of sixty-four. A small CNN like this typically reaches very high accuracy on digits within a few epochs, though your exact numbers will vary. Track validation accuracy each epoch, and evaluate on the untouched test set only once at the end.

• Adam, lr 0.001, batch 64
• Watch train vs validation

Never stop at one accuracy number. Look at the images the model gets wrong and build a confusion matrix, which we met in episode sixteen. You will often find sensible mistakes, like confusing a four with a nine, or genuinely ambiguous handwriting. Those errors tell you whether you need more data, better augmentation, or a bigger model.

• Inspect misclassified images
• Confusion matrix

More data usually helps, and augmentation fakes it. Each time an image is loaded, apply random flips, crops, small rotations or color jitter, so the network rarely sees the exact same picture twice. The label should not change: a flipped cat is still a cat. Apply augmentation to training data only, never validation or test, and be careful: flipping a digit six does not make a valid digit.

• Random flips, crops, rotations
• Applied to training data only

Training a deep CNN from scratch needs lots of data and compute. Transfer learning starts from a network already trained on a huge image dataset. Its early layers detect edges and textures useful for almost any task. You keep those layers, replace the final classification layer with one sized for your classes, and train on your own small dataset.

• Reuse pretrained features
• Replace the final layer

A reliable recipe has two stages. First freeze the pretrained layers by setting requires grad to false, and train only the new head for a few epochs. Then optionally unfreeze some later layers and train everything with a much smaller learning rate, so you adjust the features gently without destroying them. Also, preprocess your images exactly as the pretrained model expects, including its size and normalization.

• Freeze backbone, train head
• Then unfreeze with tiny lr

Finally, let us look inside. Plot the first layer's kernels as tiny images and you will often see edge detectors at various angles and simple color blobs, much like our hand-made kernel. Feed an image in and display the feature maps, and you can see which filters light up on edges, textures or specific shapes. Deeper filters are harder to interpret, so treat such visualizations as hints, not proof.

• Plot first-layer kernels
• Show feature maps

Let us recap. Tensors hold data, and autograd records operations so backward gives gradients automatically. Datasets and data loaders serve batches, and every training loop is forward, loss, zero grad, backward, step. Move model and data to the same device, save the state dict, debug by overfitting one batch, and set seeds.

• Tensors and autograd
• Dataset, DataLoader, loop

For CNNs, convolution slides a shared kernel across the image, taking dot products. Output size is floor of n plus two p minus k over s, plus one. Channels let layers learn many filters, pooling shrinks maps, and stacked layers grow receptive fields from edges to objects. Augmentation and transfer learning let you get strong results with limited data.

• Convolution = sliding dot products
• Size formula, channels, pooling

Images have grid structure, but language, audio and time series have order. In the next episode we meet sequence models: recurrent networks, LSTMs, and the attention mechanism that sets up transformers.

• Episode 28: RNNs, LSTMs, attention
• Subscribe for the full series