CNN Architectures: LeNet, ResNet, Transfer Learning and Grad-CAM Explained

A neural network can learn to tell cats from airplanes, and it does it by building a hierarchy of tiny visual ideas: edges, textures, shapes, objects. But how do we design such a network, and how do we avoid training one from scratch every time? Today we answer both questions.

• Edges, textures, shapes, objects
• Design and reuse

In lesson fifty-one we learned the mechanics of convolution: kernels, stride, padding and pooling. We will not repeat them. Today we use them. We will build and train a small CNN in PyTorch, climb from LeNet to ResNet, add data augmentation, reuse a pretrained network, and finally look inside the model with Grad-CAM.

• Build, compare, reuse, inspect
• Builds on lesson 51

One promise before we start. I will not quote accuracy numbers from papers as if they were results of our code. Any number you see in your own runs depends on your seed, your hardware and your settings. So I will describe what you should expect qualitatively, and you should trust the numbers your own experiments print.

• Numbers depend on seed and setup
• Measure, do not assume

We start with a standard dataset: CIFAR-10, sixty thousand tiny color images of thirty-two by thirty-two pixels in ten classes such as ships, frogs and trucks. Each image is a tensor with shape three by thirty-two by thirty-two. PyTorch's torchvision downloads it and wraps it in the Dataset and DataLoader tools from lesson fifty.

• 10 classes, 32 by 32 pixels
• Tensor shape 3 x 32 x 32

Our small CNN stacks three blocks. Each block is a three by three convolution, a ReLU, and a two by two max pool that halves the width and height. Channels grow as size shrinks: three, then thirty-two, sixty-four, one hundred twenty-eight. After the blocks we flatten and use a linear layer to produce ten scores, one per class.

• Conv, ReLU, pool, repeat
• Channels up, size down
• Flatten, then linear

Let's verify the shapes with numbers. With padding one, a three by three convolution keeps thirty-two by thirty-two. Pooling gives sixteen, then eight, then four. The last feature map is one hundred twenty-eight channels of four by four, which is two thousand forty-eight numbers. So the linear layer has two thousand forty-eight times ten weights, plus ten biases.

• 32, 16, 8, 4 after pooling
• 128 x 4 x 4 = 2048
• 2048 x 10 + 10 = 20,490

Training follows the loop from lesson fifty. For each mini-batch: forward pass, cross-entropy loss, zero the gradients, backward pass, optimizer step. We use Adam with a learning rate of one e minus three, and we switch the model to evaluation mode when measuring validation accuracy. A classic bug is forgetting that switch.

• Forward, loss, backward, step
• Adam, learning rate 0.001
• Remember eval mode

Plot training and validation loss per epoch. With a small CNN and no tricks, a very common pattern appears. Training loss keeps falling, while validation loss falls and then turns upward. The widening gap between them is overfitting, the generalization problem from lesson thirteen, now living inside an image model.

• Training loss keeps falling
• Validation loss turns upward

The pattern we just built is old. LeNet-5, from the nineties, read handwritten digits using convolution, pooling, and a few fully connected layers, with only tens of thousands of parameters. Its recipe of alternating convolutions and pooling, then a classifier head, is still the skeleton of most CNNs today.

• Conv, pool, conv, pool, dense
• Tens of thousands of parameters

Years later, VGG asked a simple question: what if we use only three by three convolutions and just go deeper? Stack two of them and each output pixel sees a five by five patch. Stack three and it sees seven by seven. Small kernels, many layers, and a very regular design that is easy to remember.

• Two 3x3 layers see 5x5
• Three 3x3 layers see 7x7

Count the parameters, using C channels in and out. One seven by seven convolution has forty-nine C squared weights. Three stacked three by three convolutions cover the same area with only twenty-seven C squared. That is fewer weights, plus two extra ReLUs in between, which makes the function more expressive. Fewer parameters and more nonlinearity is a great deal.

• One 7x7: 49 C squared
• Three 3x3: 27 C squared
• Extra ReLUs add power

If deeper is better, why not stack a hundred layers? People tried, and found something strange. A very deep plain network got worse, even on the training set. That is not overfitting, because overfitting would make training error low. It means the optimizer struggled to train deep stacks. This is the degradation problem.

• Plain deep nets train worse
• Not overfitting
• Optimization is the problem

ResNet's fix is beautifully simple. Instead of asking a block to learn the whole mapping H of x, we ask it to learn only a residual F of x, and then we add the input back: output equals x plus F of x. That addition is the skip connection, a shortcut that carries x around the layers untouched.

• Learn the residual F(x)
• Add the input back
• Shortcut carries x

Here is the intuition. If the best thing a block can do is nothing, a plain block must carefully learn weights that reproduce the identity, which is hard. A residual block just pushes F of x toward zero, and weights near zero are easy to reach. So adding layers can no longer hurt much, because the extra layers can fall back to doing nothing.

• Plain block must learn identity
• Residual block just outputs zero

Now the gradient. For output y equals x plus F of x, the derivative of y with respect to x is one plus the derivative of F with respect to x. That one is the key. During backpropagation the gradient always has a direct path with multiplier one, so it cannot vanish just because the derivative of F is small.

• dy/dx = 1 + dF/dx
• A direct gradient path

Let's use numbers, echoing the vanishing gradients of lesson forty-seven. Suppose each of twenty layers shrinks the gradient by a factor of zero point five. A plain stack multiplies them: zero point five to the twentieth, about one in a million. Now suppose each residual block has a factor of one plus zero point five, or one point five, on the full path but the shortcut still carries a clean one. The signal survives.

• Plain: 0.5^20 is about 1e-6
• Skip path keeps multiplier 1

A frequent bug: adding x and F of x requires the same shape. When a block changes the number of channels or halves the resolution, x no longer matches. The standard fix is a one by one convolution with the same stride on the shortcut, called a projection, so both branches agree. In PyTorch, a shape error on the plus sign usually means this.

• Add needs equal shapes
• Use a 1x1 conv projection
• Stride matches the main branch

In code, a residual block is short. Define two convolutions with batch normalization, and in forward compute out as the first convolution, batch norm, ReLU, then the second convolution and batch norm. Then return ReLU of out plus the shortcut of x. Stack a handful of these and you have a small ResNet that trains reliably much deeper than our plain network.

• Two conv plus batch norm
• Return relu(out + shortcut(x))

ResNet opened the door, and many designs followed. Some connect every layer to all earlier ones. Some use depthwise separable convolutions, which split spatial filtering and channel mixing into two cheap steps, so models run on phones. Others scale depth, width and input resolution together. The shared lesson is that structure matters as much as size.

• Depthwise separable convolutions
• Scale depth, width, resolution
• Structure beats raw size

Quick cost check with C equal to one hundred twenty-eight channels in and out. A standard three by three convolution has nine times one twenty-eight times one twenty-eight, about one hundred forty-seven thousand weights. The separable version needs nine times one twenty-eight for the depthwise step, plus one twenty-eight squared for the pointwise step, about seventeen and a half thousand. Roughly eight times fewer.

• Standard: 9 x 128 x 128
• Separable: 9 x 128 + 128 x 128
• About 8 times fewer

Back to our overfitting curves. The cheapest cure is more data, and when we cannot collect any, we can fabricate variations. A flipped cat is still a cat. A slightly shifted ship is still a ship. Data augmentation applies random label-preserving changes to each image every epoch, so the network almost never sees exactly the same picture twice.

• Flip, crop, color jitter
• Same label, new pixels

Three transforms cover most small projects. Random horizontal flip mirrors the image half of the time. Random crop pads the image by a few pixels and cuts a random thirty-two by thirty-two window, shifting the object slightly. Color jitter nudges brightness, contrast and saturation. In torchvision you chain them with Compose and apply them to the training set only.

• Compose transforms in torchvision
• Apply to training set only

This is a mistake worth underlining. Validation and test sets must be left clean, apart from normalization. Their job is to measure performance on real data, and random changes would add noise to the measurement. Also choose transforms that respect the task: flipping is fine for animals, but a flipped digit six is not a six anymore.

• Augment training only
• Respect task semantics

Rerun the same network with augmentation. What do you expect to see? Training loss falls more slowly, because the task is harder. Validation loss usually stays low for longer, and the gap between the curves shrinks. Often training takes more epochs. Check this in your own runs, since the size of the benefit depends on your model and data.

• Training improves more slowly
• Validation holds up longer
• Verify in your runs

Even with augmentation, training from scratch needs plenty of data. But remember what CNNs learn: early layers find edges and colors, middle layers find textures and parts. Those are useful for almost any photographs. So we can take a ResNet pretrained on a large dataset, ImageNet, which has over a million labelled images, and reuse its knowledge. This is transfer learning.

• Early layers: edges and colors
• Pretrained on ImageNet
• Reuse for a new task

Option one is feature extraction. Load a pretrained ResNet, freeze every parameter by setting requires grad to false, and replace the last linear layer with a new one that outputs your number of classes. Only that new head trains. It is fast, needs little data, and cannot damage the pretrained weights.

• requires_grad = False
• New final linear layer
• Train only the head

Option two is fine-tuning: unfreeze some or all layers and keep training the whole network on your data. This lets features adapt to your domain, which helps when your images differ from ImageNet, like medical scans. It needs more data and more care, because large updates can wreck the pretrained weights in a few steps.

• Update pretrained layers too
• Needs more data and care

The learning rate is where fine-tuning succeeds or fails. A common recipe is a normal rate, like one e minus three, for the fresh head, and a much smaller rate, like one e minus five or one e minus four, for the pretrained body. In PyTorch you pass two parameter groups to the optimizer, each with its own learning rate. Treat these values as starting points to tune.

• Head: larger rate, about 1e-3
• Body: much smaller rate
• Use parameter groups

Pretrained models expect inputs prepared the way they were trained. That means resizing to the expected size, often two hundred twenty-four by two hundred twenty-four, and normalizing with the ImageNet channel means and standard deviations. Forgetting this is a silent bug: the code runs, but the features are fed unfamiliar values and results are mysteriously poor. Also keep batch norm layers in a sensible mode when freezing.

• Resize as the model expects
• Use ImageNet mean and std
• Silent bug: wrong inputs

Compare three runs in your own notebook: scratch, feature extraction, fine-tuning. Typically the pretrained runs reach good validation performance in far fewer epochs, and fine-tuning often edges ahead given enough data. With very small datasets, feature extraction can be safer. Record all three on the same split, with the same seed, so the comparison is fair.

• Scratch, extract, fine-tune
• Same split, same seed

Now let's look inside. The first layer of a CNN has kernels that act directly on pixels, so we can show them as tiny color images. In a trained network you typically see oriented edge detectors, color blobs and simple stripe patterns. Deeper kernels have many input channels and are much harder to interpret directly, so we need other tools.

• Kernels as tiny images
• Edges, colors, stripes

A better view is the feature map: the output of a layer for one specific image. Register a forward hook on a layer, run an image through, and plot each channel as a grayscale picture. Early maps look like the photo with edges highlighted. Later maps are smaller and more abstract, lighting up where a certain part or texture appears.

• Layer output for one image
• Use forward hooks
• Early: edges; late: parts

Grad-CAM answers a sharper question: which regions of the image made the model choose this class? Take the last convolutional feature maps. Compute the gradient of the class score with respect to each map. Average each gradient over space to get one weight per channel, saying how important that channel is for this class.

• Gradient of class score
• Average over space
• One weight per channel

Then combine: multiply each feature map by its weight, add them up, and keep only positive values with a ReLU. In symbols, the heatmap is ReLU of the sum over k of alpha k times A k. Upsample the small map to the image size and overlay it. Bright areas are the evidence for the class.

• Heatmap = ReLU(sum of alpha_k A_k)
• Upsample and overlay

Use it as a debugging tool. If a model labels a boat correctly but the heat sits on the water, it may have learned the background rather than the object. That is a shortcut. Grad-CAM is coarse and only a clue, not proof of reasoning, but when the attention falls on the wrong place, it tells you what to fix in the data.

• Heat on object: good sign
• Heat on background: shortcut
• A clue, not proof

Accuracy hides where the model fails. Build a confusion matrix on the validation set: rows are true classes, columns are predictions. The diagonal is correct. Large off-diagonal cells name the systematic mistakes. For CIFAR-ten the usual suspects are visually similar pairs, like cats and dogs, or trucks and automobiles. Check yours.

• Rows true, columns predicted
• Off-diagonal cells matter

Now go beyond the matrix and view the actual images. Collect the most confident wrong predictions and plot them with their true label, predicted label and confidence. You will often find blurry photos, odd crops, ambiguous objects, or even wrong labels in the dataset. Each category suggests a different fix: better augmentation, more data, a cleaner dataset, or a stronger model.

• Plot top wrong predictions
• Find the cause, then fix

Let's recap. A CNN stacks convolution, ReLU and pooling blocks with growing channels. VGG showed that small three by three kernels, stacked deep, are efficient. ResNet added the skip connection, output equals x plus F of x, which gives gradients a path with multiplier one. Augmentation narrows the validation gap, and it belongs on the training set only.

• Small kernels, deep stacks
• y = x + F(x)
• Augment train only

Transfer learning reuses pretrained features: freeze and replace the head for small data, or fine-tune with a small learning rate for the body and a larger one for the head. Match the preprocessing. Then inspect: filters, feature maps and Grad-CAM show what the network uses, and a confusion matrix plus wrong examples show where it breaks.

• Freeze or fine-tune
• Grad-CAM and confusion matrix

Images have spatial structure, but a lot of data arrives in order: sentences, audio, sensor streams. In lesson fifty-three we turn to LSTMs, GRUs and backpropagation through time, and see how gates tame the vanishing gradients that plague simple recurrent networks.

• Lesson 53: gates and BPTT
• Subscribe for the series