Derivatives From First Principles: Slopes, Rules and Why ML Needs Them

A model has a million knobs, and it is wrong. Which way do you turn each knob to make it less wrong? The answer is a single idea from calculus: the derivative.

• A model is wrong
• Which direction improves it?

Last time we studied eigenvectors, SVD and positive-definite matrices. Today we start calculus, and we build the derivative from first principles, with no memorized rules.

• Last time: eigenvectors and SVD
• Today: calculus begins

A function takes an input and gives an output, and its graph draws every pair as a point. For the function f of x equals x squared, an input of two gives four, and an input of three gives nine.

• Input in, output out
• Graph shows every pair

A limit asks what value a function approaches as the input gets closer and closer to a point, even if it never lands there. Continuity means the graph has no jumps or holes, so the limit equals the actual value.

• Limit: approaching a value
• Continuous: no jumps or holes

Average rate of change is rise over run between two points. On the curve x squared, going from x equals one to x equals three, the rise is eight over a run of two, so the slope is four.

• Rise over run
• Secant line through two points

Now slide the second point closer to the first. The secant line pivots, and as the gap h shrinks toward zero, it settles into the tangent line, which touches the curve at just one point. Its slope is the instantaneous rate of change.

• Shrink the gap h
• Tangent slope = instantaneous rate

This gives the definition. The derivative of f at x is the limit, as h goes to zero, of f of x plus h minus f of x, all divided by h.

• f'(x) = lim (f(x+h) − f(x)) / h
• h shrinks to zero

Try it on x squared. Replace f of x plus h with x plus h, squared, which expands to x squared plus two x h plus h squared. Subtract x squared, divide by h, and you get two x plus h. As h goes to zero, the derivative is two x.

• Expand (x+h)²
• Cancel, divide by h
• Let h → 0: slope is 2x

The same pattern gives the power rule. Expanding x plus h to the n, the first two terms are x to the n and n times x to the n minus one times h. Every other term has at least h squared, so it vanishes. The derivative of x to the n is n times x to the n minus one.

• (x+h)ⁿ = xⁿ + n·xⁿ⁻¹·h + …
• Higher terms vanish
• d/dx xⁿ = n·xⁿ⁻¹

For a product, picture a rectangle with sides f and g. If both sides grow slightly, the area gains a strip along each side, plus a tiny corner that vanishes in the limit. So the derivative of f times g is f prime g plus f g prime.

• Area of a rectangle fg
• (fg)′ = f′g + fg′

The quotient rule follows from the product rule. Write f over g as f times g to the minus one, differentiate with the product rule and the chain rule, and tidy up. You get f prime g minus f g prime, all over g squared.

• Write f/g as f·g⁻¹
• (f/g)′ = (f′g − fg′) / g²

The chain rule handles functions inside functions. If y depends on u, and u depends on x, a small change in x changes u, which changes y. The rates multiply: dy by dx equals dy by du times du by dx. This rule is the heart of backpropagation.

• Rates multiply along the chain
• dy/dx = dy/du · du/dx

The number e is special because the function e to the x has a slope equal to its own height at every point. Using the limit definition, e to the x times e to the h minus one over h, and that last factor tends to one. So the derivative of e to the x is e to the x.

• Slope equals height
• d/dx eˣ = eˣ

The natural log is the inverse of e to the x. Differentiate both sides of e to the ln x equals x using the chain rule, and you find the derivative of ln x is one over x. The slope is steep near zero and gentle far away.

• Inverse of eˣ
• d/dx ln x = 1/x

In Python we can approximate a derivative numerically with a tiny h. Using the symmetric difference, f of x plus h minus f of x minus h, over two h, is accurate and simple. But h too small causes floating point round-off error.

• Symmetric difference quotient
• Too small an h hurts accuracy

Symbolic tools like SymPy apply the rules exactly and return formulas, such as two x. Numerical methods are quick and general, but approximate. Machine learning libraries use a third approach, automatic differentiation, which we meet later.

• Symbolic: exact formula
• Numerical: quick approximation

Integration is the reverse idea. It measures the area under a curve by adding thin rectangles, and as they get thinner the sum becomes exact. We will need this later, because the area under a probability curve is a probability.

• Add up thin rectangles
• Area = probability, later

Here is the machine learning link. Imagine a loss curve over one parameter. If the slope at the current point is positive, the loss rises as the parameter grows, so we move the parameter left. If the slope is negative, we move it right.

• Positive slope: move left
• Negative slope: move right

Written as an update, the new parameter equals the old parameter minus a learning rate times the slope. A steeper slope means a bigger step, and a flat slope means we have reached a minimum. That one line is gradient descent in one dimension.

• w ← w − α · slope
• Flat slope means minimum

Let us recap. The derivative is the limit of the secant slope as the gap shrinks to zero. We derived x squared, the power, product, quotient and chain rules, and the derivatives of e to the x and ln x. Integration is area, and slope tells a model which way to move.

• Derivative = limit of secant slope
• Rules derived, not memorized
• Slope guides parameter updates

Next time we add more variables, with partial derivatives, gradients, Jacobians and Hessians.

• Partial derivatives and gradients
• Subscribe for episode 7