PyTorch Tensors and Autograd: Gradients You Can Verify by Hand

In lesson 36 we differentiated a computational graph by hand, node by node. It worked, but imagine doing that for a network with a hundred million parameters. Today you meet the tool that does it for you, exactly and automatically. And we will not just trust it. We will prove it right.

• Gradients drive all learning
• Autograd computes them for us

Here is the plan. First, tensors: how to create them, their dtype, their device, and how they relate to NumPy from lesson 31. Then autograd: the dynamic graph, backward, and the controls for turning gradients on and off. Finally we fit a linear regression using nothing but raw tensors, and we finish with the classic bugs. We skip data loaders and training loop structure, which come next lesson.

• Tensors and shapes
• Autograd and the graph
• Linear regression, raw tensors

A tensor is simply an n-dimensional array of numbers, the same object you met as a NumPy array in lesson 31. A single number is a zero-dimensional tensor, a list is one-dimensional, a table is two-dimensional, and a stack of tables is three-dimensional. What PyTorch adds is two superpowers: it can live on a graphics card, and it can remember how it was computed.

• Scalar, vector, matrix, tensor
• Same idea as NumPy arrays

You create a tensor by calling torch dot tensor on a Python list. There are also factories: zeros and ones take a shape, rand draws uniform numbers, randn draws from a standard normal, arange counts, and linspace spaces points evenly. Calling torch dot zeros with arguments two and three gives a grid with two rows and three columns, all zero. Each of these is a one-line replacement for a loop you would otherwise write.

• torch.tensor, zeros, ones, rand
• arange and linspace

Every tensor has a dtype. Floating-point numbers default to thirty-two bit floats, while a list of whole numbers becomes sixty-four bit integers. This matters because gradients only exist for floating-point tensors, and because multiplying an integer tensor by a float tensor in certain operations raises an error. When in doubt, build your tensor with decimal points, or convert with the to method and a dtype.

• float32 is the default
• Integers are int64
• Mismatches cause errors

The device says where the tensor lives: on the CPU in ordinary memory, or on a GPU. You move a tensor with the to method. The rule is strict: every tensor in an operation must live on the same device, otherwise PyTorch complains. We will use GPUs properly next lesson. For today everything stays on the CPU, which is plenty for small examples.

• Tensors live on a device
• Operands must share a device

Shape operations rearrange the same numbers without changing them. Reshape turns a tensor with twelve elements into three by four, or two by six, as long as the count matches. Transpose swaps two axes, unsqueeze inserts an axis of size one, and squeeze removes one. A handy trick is using minus one in reshape, which tells PyTorch to work out that dimension for you.

• reshape and view
• transpose and permute
• squeeze and unsqueeze

Broadcasting works exactly as in lesson 31. Line up the shapes from the right. Two sizes are compatible if they are equal or one of them is one. Take a column of shape three by one and a row of shape one by four. The column stretches across four columns, the row stretches down three rows, and the result has shape three by four.

• Align shapes from the right
• Sizes must match or be 1

Here is the most expensive broadcasting mistake. Suppose predictions have shape three, and targets have shape three by one. Subtract them and you do not get three errors. You get a three by three grid of every prediction minus every target, and your loss is quietly wrong. No error message appears. Always print shapes, and make predictions and targets match before subtracting.

• Shape (3) minus shape (3,1)
• Result is (3,3), not (3)

Moving between NumPy and PyTorch is cheap. Torch dot from underscore numpy builds a tensor that shares memory with the array, so changing one changes the other. The numpy method on a tensor does the reverse. If you want independence, make a copy. This sharing is fast, but it is also a source of surprising bugs when you forget that two names point at the same numbers.

• from_numpy shares memory
• clone makes an independent copy

Now the magic. When you create a tensor with requires_grad set to true, PyTorch starts recording every operation that touches it. We call such inputs leaf tensors: the parameters at the edge of the graph. Think of it as pressing record on a camera. Any result computed from a recorded tensor is itself recorded, so a trail of history grows behind every calculation.

• Leaf tensors we want gradients for
• Set requires_grad=True

PyTorch builds the computational graph while your code runs, line by line. This is called a dynamic graph. Each arithmetic operation adds a node. If your code has an if statement or a loop, the graph simply follows whatever path actually executed. The graph is thrown away after use and rebuilt on the next forward pass, which makes debugging feel like ordinary Python.

• Graph built as code runs
• Rebuilt on every forward pass

Each tensor produced by a recorded operation carries an attribute called grad_fn. It names the operation that created it, like multiplication backward or power backward, and knows how to push gradients to its inputs. Leaf tensors you created yourself have grad_fn equal to none. Following the chain of grad_fn objects from the loss back to the leaves retraces the graph.

• Every result stores grad_fn
• Leaf tensors have none

Calling backward on a scalar loss walks the graph in reverse, applying the chain rule from lesson 36 at every node. When it finishes, each leaf tensor has a dot grad attribute holding the derivative of the loss with respect to it. The loss has to be a single number, because a gradient needs one output to differentiate. For a vector output, you first reduce it, for example by taking a sum or a mean.

• Loss must be a scalar
• Gradients land in .grad of leaves

Let us rebuild a graph like the one from lesson 36. Let x equal two, w equal three, b equal one, and target t equal ten. Then z equals w times x plus b, which is seven. The error is z minus t, which is minus three, and the loss is that squared: nine. By hand, the chain rule says the derivative of the loss with respect to z is two times minus three, equal to minus six.

• z = w·x + b
• L = (z − t)²
• x=2, w=3, b=1, t=10

Now push that minus six backward. The derivative with respect to w is minus six times x, which is minus twelve. With respect to b it is minus six times one, which is minus six. With respect to x it is minus six times w, which is minus eighteen. These three numbers are our ground truth, computed purely on paper.

• dL/dw = −6 · 2 = −12
• dL/db = −6 · 1 = −6
• dL/dx = −6 · 3 = −18

In PyTorch we create w, b and x with requires_grad true, compute the loss, and call backward. Printing the grad attributes gives minus twelve for w, minus six for b, and minus eighteen for x. They match our hand derivation exactly. Autograd is not a different mathematics. It is the same chain rule from lesson 36, applied mechanically at every node.

• w.grad = -12
• b.grad = -6
• x.grad = -18

There is a second, independent check that needs no calculus at all. Nudge w by a tiny epsilon, say one thousandth, and watch how the loss moves. At w equals three point zero zero one, the loss is eight point nine eight eight. At two point nine nine nine, it is nine point zero one two. The difference, minus point zero two four, divided by two point zero zero two, gives minus twelve. Three methods, one answer.

• Nudge w by epsilon
• Slope ≈ (L(w+ε) − L(w−ε)) / 2ε

This finite-difference trick is the standard gradient check, and you should use it whenever you write a new operation yourself. Use sixty-four bit floats and small inputs, because with tiny epsilons, thirty-two bit rounding noise can swamp the signal. Finite differences are too slow for training, since they need two forward passes per parameter. But as a test, they are priceless.

• Gradient checks catch bad code
• Use float64 and small tensors

Here is a behaviour that surprises beginners. Calling backward does not overwrite dot grad, it adds to it. Run backward a second time on a fresh loss from the same parameters, and w dot grad becomes minus twenty-four instead of minus twelve. Accumulation is useful in advanced cases, such as simulating large batches, but by default it is a trap.

• backward adds into .grad
• Second call: −24

The cure is to reset gradients before each new backward pass. For a raw tensor you can call its grad dot zero underscore, or set its grad to none. Later, optimizers wrap this in a method also called zero_grad. The rhythm of every training step is the same: compute loss, clear old gradients, backward, update. Forgetting the clearing step makes gradients from different steps pile up and your updates become nonsense.

• Zero grads before each backward
• Or set grad to None

Sometimes you do not want the camera rolling. Inside a with torch dot no_grad block, operations are not recorded, so no graph is built and memory is saved. You need this when you update parameters, because the update itself is not part of the model, and when you evaluate a model without training. Without it, PyTorch would try to differentiate through your update step.

• Disables graph building
• Use for updates and evaluation

The detach method returns a tensor with the same values but no history. It cuts the graph at that point, so gradients will not flow past it. You use it to log a loss value, convert to NumPy, or deliberately treat one quantity as a constant. Remember that the detached tensor still shares memory with the original, so modifying it changes the original numbers too.

• Same data, no history
• Shares memory with original

Occasionally you need an operation PyTorch does not know how to differentiate, or you want a more stable gradient. You can subclass torch dot autograd dot Function and write two static methods. Forward computes the output and saves what it needs. Backward receives the gradient of the loss with respect to the output and returns the gradient with respect to each input.

• Subclass torch.autograd.Function
• Define forward and backward

Take the square function. Forward returns x squared and saves x. Backward multiplies the incoming gradient by two times x, which is just the derivative from lesson 6 combined with the chain rule. At x equals three with an incoming gradient of one, backward returns six. Check it with the finite-difference test and you are done. Most of the time you will never need this, but it demystifies how every built-in operation works.

• forward: y = x²
• backward: grad × 2x

Now we put everything together, with no nn dot Module in sight. Our data has four points: x equals zero, one, two, three, and y equals one, three, five, seven. The hidden truth is y equals two x plus one. We model predictions as w times x plus b, start both parameters at zero, and mark them with requires_grad true. Broadcasting handles all four points at once.

• x = 0, 1, 2, 3
• y = 1, 3, 5, 7
• Model: ŷ = w·x + b

With both parameters zero, every prediction is zero. The errors, prediction minus target, are minus one, minus three, minus five, minus seven. Squared, they are one, nine, twenty-five, forty-nine, which add up to eighty-four. Divide by four and the mean squared error is twenty-one. That is a big loss, which is exactly what we expect for a flat line far from the data.

• Predictions all 0
• Errors: −1, −3, −5, −7
• MSE = 84 / 4 = 21... check

By the formulas from lesson 14, the gradient with respect to w is two over n times the sum of error times x. That sum is zero plus minus three plus minus ten plus minus twenty-one, equal to minus thirty-four, and half of that is minus seventeen. For b, the errors sum to minus sixteen, giving minus eight. Calling backward on the loss in PyTorch produces exactly these two numbers.

• dL/dw = (2/n) Σ err·x = −17
• dL/db = (2/n) Σ err = −8

Now we step downhill with a learning rate of point one. The new w is zero minus point one times minus seventeen, which is one point seven. The new b is zero minus point one times minus eight, which is point eight. Recompute: predictions are point eight, two point five, four point two, five point nine. Errors are minus point two, minus point five, minus point eight, minus one point one, and the loss falls from twenty-one to point five three five.

• Learning rate 0.1
• w = 0 − 0.1·(−17) = 1.7
• b = 0.8

In code, the update is done inside a no_grad block, using in-place subtraction on w and b. Right after, we zero their gradients. That in-place update is allowed because we turned recording off. Repeat this cycle a few hundred times and w approaches two while b approaches one, recovering the hidden rule. We wrote the whole learning algorithm from raw tensors and autograd.

• Update in place inside no_grad
• Then zero the gradients

The first common bug: modifying a leaf tensor in place while it requires gradients. Writing w minus-equals something outside no_grad raises a runtime error saying a leaf variable that requires grad is being used in an in-place operation. PyTorch protects you because that change would corrupt the recorded history. The fix is to perform the update inside no_grad. In-place operations on intermediate tensors can also break backward if a value was saved for later.

• Error: leaf variable modified in-place
• Fix: use no_grad for updates

The second bug is forgetting to clear gradients. In our example, the second step would use minus seventeen plus the new gradient, not the new gradient alone. Steps grow too large, the loss may bounce or explode, and nothing crashes to warn you. If a training run behaves strangely from the very second step, check this first.

• Gradients pile up
• Steps overshoot or diverge

The third bug involves graph retention. By default, backward frees the graph, so calling it twice on the same loss raises an error unless you pass retain_graph true. The opposite problem is more common: storing the loss tensor itself in a list for logging keeps the entire graph alive, and memory climbs every step. Store loss dot item, or loss dot detach, instead, which keeps only the number.

• backward frees the graph
• Keeping loss tensors leaks memory

Let us recap. Tensors are NumPy-like arrays with a dtype, a device, and the same broadcasting rules. Setting requires_grad builds a dynamic graph, each result stores a grad_fn, and backward fills dot grad on the leaves. We verified a hand-derived graph with autograd and with finite differences, and we fit a line using raw tensors. Remember to zero gradients, update under no_grad, and detach what you log.

• Tensors: shape, dtype, device
• Autograd: graph, backward, grad
• Controls: zero_grad, no_grad, detach

In the next lesson, lesson 50, we turn this hand-rolled loop into a proper training pipeline with datasets, data loaders, a clean training loop, GPUs, and debugging tools. See you there.

• Datasets and DataLoader
• Training loops, GPUs, debugging