Partial Derivatives, Gradients, Jacobians and Hessians Explained Visually

A neural network can have millions of inputs to its loss, all at once. How do you take a derivative of that? The answer is a small set of tools: partial derivatives, gradients, Jacobians, and Hessians.

• Millions of inputs
• One tool kit

Last time, we defined the derivative from first principles as the slope of a curve at a point. Today we upgrade that idea to functions with several inputs, and we finish by setting up backpropagation.

• Derivative equals slope
• Now: many inputs

A function of two variables, like f of x and y equals x squared plus y squared, takes two numbers and returns one. Plot the output as height, and the function becomes a surface over the xy plane.

• Two inputs, one output
• Output becomes height

Now ask how steep it is. That depends on direction. So we freeze one variable and nudge the other. That is a partial derivative: the ordinary derivative, holding everything else constant.

• Hold other variables fixed
• Slope along one axis

Let us do one by hand. Take f of x and y equals x squared times y, plus three y. For the partial with respect to x, treat y as a constant. The derivative of x squared times y is two x y, and three y vanishes. So the answer is two x y.

• f = x²y + 3y
• ∂f/∂x = 2xy

Now the other way. Treat x as constant. The derivative of x squared times y with respect to y is just x squared, and the derivative of three y is three. So the partial with respect to y is x squared plus three.

• Treat x as constant
• ∂f/∂y = x² + 3

Stack the partial derivatives into one vector and you get the gradient. Written with an upside-down triangle called nabla, it is the list of all partials, one per input.

• Vector of all partials
• One entry per input

On a contour plot, which is a map of equal-height lines, the gradient has a clear meaning. It points in the direction of steepest ascent, and it is always perpendicular to the contour line through that point.

• Points to steepest ascent
• Perpendicular to contours

The length of the gradient tells you how steep that steepest slope is. Near the bottom of the bowl the arrows shrink to nothing. A zero gradient means a flat spot: a minimum, a maximum, or a saddle.

• Length equals steepness
• Zero: min, max, or saddle

What if you walk in some other direction, given by a unit vector u? The directional derivative is the dot product of the gradient and u. This is the dot product from our vectors episode, measuring alignment.

• Slope = ∇f · u
• Max when aligned with gradient

That explains steepest ascent. The dot product is largest when u points along the gradient, equal to the gradient's length. Walk perpendicular and the slope is zero. Walk opposite and you descend as fast as possible.

• Aligned: slope equals length
• Perpendicular: zero slope

The gradient handles one output. But a function can also return several outputs, like a layer in a neural network mapping a vector to a vector. Then each output has its own gradient.

• Vector in, vector out
• Each output has a gradient

Stack those gradients as rows and you get the Jacobian matrix. Entry in row i and column j is the partial derivative of output i with respect to input j. Its shape is outputs by inputs.

• Rows: outputs, columns: inputs
• Shape: outputs by inputs

Geometrically, near a point the Jacobian is the best linear approximation of the function. It acts as the matrix transformation from episode four, turning a tiny square of input into a tiny parallelogram of output.

• Best local linear map
• Square becomes parallelogram

The gradient tells slope. To see curvature, we take derivatives of the derivatives. Arrange all second partial derivatives into a square matrix, and you get the Hessian.

• Second partials in a matrix
• Square and symmetric

For f equals x squared plus three y squared, the Hessian is the diagonal matrix with two and six. Curvature is stronger along y than along x, so the bowl is narrower in that direction, and the contours are ellipses.

• f = x² + 3y²
• H = diag(2, 6)

Remember eigenvalues from episode five. The Hessian's eigenvalues are the curvatures along its eigenvector directions. If all are positive, the Hessian is positive definite, and the point is a bowl: a true local minimum.

• All positive: bowl
• Positive definite: minimum

If the eigenvalues have mixed signs, the surface curves up one way and down another. That is a saddle point. All negative gives a dome. So the Hessian classifies every flat spot the gradient finds.

• Mixed signs: saddle
• All negative: maximum

Real models are long chains of simple operations. We draw them as computational graphs: nodes for operations, arrows for data flowing forward. Take a tiny example, f equals x times y, plus z.

• Nodes: operations
• Edges: data flow

The chain rule says that when one quantity depends on another through a middle step, you multiply the local derivatives along the path. If f depends on a, and a depends on x, then df by dx equals df by da times da by dx.

• Multiply local derivatives
• Along each path

Apply it backward. At the add node, df by da is one and df by dz is one. At the multiply node, da by dx is y, which is three, and da by dy is x, which is two. Multiply along paths: df by dx is three, df by dy is two, df by dz is one.

• Local derivatives at each node
• Result: (3, 2, 1)

This backward sweep, reusing local derivatives to get the whole gradient in one pass, is backpropagation. We will build it in full when we reach neural networks. Today the point is the idea.

• Forward pass, backward pass
• Full version in episode 26

How can we be sure our derivatives are right? Use finite differences. Nudge an input by a tiny number, like ten to the minus five, and measure how much the output changes, divided by the nudge.

• Nudge input by tiny h
• Central difference is more accurate

In Python, loop over each input. Add h to one coordinate, subtract h, evaluate the function both times, and divide the difference by two h. Compare with your analytic gradient. For our example, both give three, two, and one, up to tiny rounding error.

• Loop over each coordinate
• Compare analytic and numeric

This gradient check is a standard debugging tool. Finite differences are too slow for training, but perfect for catching mistakes in hand-derived or hand-coded gradients.

• Slow but trustworthy
• Catches gradient bugs

Let us recap. Partial derivatives measure slope along one input. The gradient stacks them and points uphill, perpendicular to contours. The directional derivative is the gradient dotted with a direction.

• Partials, gradient
• Directional derivative

The Jacobian holds all partials of a vector function, and the Hessian holds second partials, revealing bowls, domes, and saddles through eigenvalues. Computational graphs and the chain rule give us the backward pass, and finite differences check our work.

• Jacobian and Hessian
• Chain rule, finite checks

Next episode we switch to probability: events, conditional probability, and Bayes' theorem. See you there.

• Subscribe for the series
• Episode 8: Bayes' theorem