Loss Functions and Gradient Descent Explained Visually, With Numbers

Every machine learning model you will ever meet, from a tiny line fit to a giant language model, learns the same way. It makes a guess, measures how wrong the guess was, and nudges itself to be a little less wrong. Repeat that millions of times, and you get intelligence of a kind. Today we build that loop from scratch.

• Guess
• Measure error
• Nudge and repeat

In the last episode we met maximum likelihood, entropy and cross-entropy, which gave us a principled way to say how well a model explains data. Now we turn that idea into an engine. We will define loss, compare two classic loss functions by hand, derive gradient descent, and watch it succeed and fail on animated landscapes.

• Loss and cost
• Derive the update rule
• Learning rate, momentum, Adam

Start with the objective function. It is a single number that says how good or bad a choice of model settings is. In machine learning we usually want that number as small as possible, so we call it a loss when it measures the error on one example, and we minimize it. Everything else in this video is a method for pushing that number down.

• One number scores a model
• Smaller means better
• Learning means minimizing it

Loss versus cost is mostly a naming convention. The loss is the penalty on a single example. The cost, sometimes called the empirical risk, is the average loss over the whole training set. Many people use the words interchangeably, so always check what the author means. In symbols, the cost J equals one over n times the sum, from i equals one to n, of the loss on example i.

• Loss: one example
• Cost: average over data
• J = (1/n) Σ loss

Next, two words that beginners mix up. Parameters are the numbers the model learns from data, such as the weight w and bias b in a line. Hyperparameters are the knobs you set before training, such as the learning rate, the batch size, or the number of training steps. The optimizer adjusts parameters. You adjust hyperparameters.

• Parameters: learned from data
• Hyperparameters: set by you
• Optimizer tunes parameters

Here is the picture to keep in mind. Imagine the loss as a landscape. Each point on the ground is one choice of parameters, and the height above that point is the loss. Training means walking across the ground until you reach a low spot. The hyperparameters decide how you walk, and the parameters are where you stand.

• Ground position = parameters
• Height = loss
• Training = walking downhill

Now let us compute two famous losses by hand. Mean absolute error, MAE, averages the absolute size of each error. Mean squared error, MSE, averages the square of each error. With y as the true value and y hat as the prediction, the error is y minus y hat. MAE takes its absolute value, and MSE squares it.

• Error = y minus y hat
• MAE: absolute value
• MSE: square

Take four true values: ten, twelve, fourteen and sixteen. A model predicts eleven, eleven, fourteen and fifteen. The errors are minus one, plus one, zero and plus one. MAE is the absolute errors, one plus one plus zero plus one, which is three, divided by four, so zero point seven five. MSE is one plus one plus zero plus one, again three, divided by four, also zero point seven five.

• Errors: -1, 1, 0, 1
• MAE = 0.75
• MSE = 0.75

They tie, because every error has size at most one, and squaring a number of size one changes nothing. Now we introduce an outlier. Change the last prediction from fifteen to twenty-six, so that last error is ten. The absolute errors are one, one, zero and ten, summing to twelve, so MAE is three.

• Last prediction becomes 26
• Last error is 10
• MAE = 12 / 4 = 3

Now MSE. The squared errors are one, one, zero and one hundred. They sum to one hundred and two, and dividing by four gives twenty-five point five. MAE rose from zero point seven five to three, a factor of four. MSE jumped from zero point seven five to twenty-five point five, a factor of thirty-four. One bad point took over the whole average.

• Squared errors: 1, 1, 0, 100
• MSE = 102 / 4 = 25.5
• One point dominates

Why does squaring punish big errors so much? Plot the penalty against the error. The absolute loss is a V shape that grows at a steady rate. The squared loss is a parabola that gets steeper the farther out you go. Doubling an error doubles the absolute penalty but quadruples the squared penalty.

• Absolute: steady growth
• Squared: steeper and steeper
• Double error, four times penalty

The gradients tell the same story. The derivative of squared error with respect to the prediction is proportional to the error itself, so a big miss produces a big push and a tiny miss produces a gentle one. The absolute error has a constant slope of plus or minus one, no matter how far off you are. That constant slope is robust to outliers, but it is not smooth at zero.

• Squared: slope grows with error
• Absolute: slope always ±1
• Absolute has a kink at zero

So which should you use? MSE is smooth, easy to differentiate, and ties naturally to the normal distribution through maximum likelihood, which we saw last time. But it is sensitive to outliers. MAE is more robust and its minimizer is the median instead of the mean, but its gradient is less informative near zero. Huber loss blends both: quadratic for small errors, linear for large ones.

• MSE: smooth, outlier-sensitive
• MAE: robust, kinked
• Huber: best of both

We know how to measure error. Now, how do we reduce it? Suppose the cost is L of theta, where theta stands for all the parameters. We stand at a point and want to take a small step delta that lowers the cost as much as possible. We only know the local slope, so we use the first-order approximation from calculus.

• Know only the local slope
• Approximate the curve by a line
• L(θ+Δ) ≈ L(θ) + ∇L·Δ

Look at that formula. The change in cost is approximately the dot product of the gradient with the step. From the vectors episode, a dot product is most negative when the two vectors point in exactly opposite directions. So the steepest decrease comes from stepping directly against the gradient. That is the whole insight.

• Change ≈ gradient · step
• Most negative when opposite
• Steepest descent direction

We choose the step to be minus eta times the gradient, where eta is a small positive number called the learning rate. Plug this in and the predicted change in cost becomes minus eta times the gradient dotted with itself, which is minus eta times the squared length of the gradient. That is never positive. So the update rule is: new theta equals old theta minus eta times the gradient of L at old theta.

• Δ = −η ∇L
• Cost change ≈ −η ‖∇L‖²
• θ ← θ − η ∇L(θ)

Here is a worked example with one parameter. Let the loss be w minus three, squared. Its derivative is two times w minus three. Start at w equals zero with learning rate zero point one. The gradient is minus six, so w becomes zero minus zero point one times minus six, which is zero point six. Next, the gradient is minus four point eight, so w becomes one point zero eight.

• L(w) = (w − 3)²
• Gradient: 2(w − 3)
• w: 0 → 0.6 → 1.08

Notice the pattern. The next gradient is minus three point eight four, and w becomes one point four six four. Each step shrinks the distance to the minimum by the same factor. The error, three minus w, goes from three to two point four, to one point nine two, to one point five three six. That factor is one minus two eta, which equals zero point eight. The steps get smaller as the slope flattens, so the walk slows down by itself near the bottom.

• Error: 3, 2.4, 1.92, 1.536
• Factor = 1 − 2η = 0.8
• Steps shrink near the bottom

The learning rate is the most important dial in optimization. For this same bowl, the error is multiplied by one minus two eta every step. If eta is small, that factor is close to one and progress crawls. If eta is exactly zero point five, the factor is zero and we land on the minimum in a single step. Between zero and one, the error shrinks smoothly.

• Error × (1 − 2η) each step
• η = 0.5 lands in one step
• Small η crawls

Now push eta higher. At eta equal to one, the factor is minus one. Starting at zero, w jumps to six, then back to zero, then to six, forever. We overshoot by exactly as much as we started. At eta equal to one point one, the factor is minus one point two, so each jump is larger than the last, and the parameter explodes. That is divergence.

• η = 1: bounces forever
• η = 1.1: factor −1.2
• Divergence: loss explodes

Let us see all four regimes on a contour plot of a stretched bowl, where the loss is x squared plus ten times y squared. Contour lines are tight in the y direction and wide in the x direction. A small learning rate marches steadily but slowly along the shallow x direction. A good rate converges quickly. A rate that is too big zigzags across the narrow valley, and one beyond the limit shoots outward.

• Too small: slow crawl
• Good: smooth convergence
• Too large: zigzag
• Beyond limit: divergence

The same four stories on a three dimensional surface. Picture the bowl as a trough with a steep wall in the y direction and a gentle slope along x. The gradient is two x and twenty y. Stability requires eta below one tenth, because the y direction multiplies its error by one minus twenty eta. The steepest direction sets the speed limit for the whole walk.

• Gradient: (2x, 20y)
• y factor: 1 − 20η
• Steepest direction limits η

A common mistake is to watch only the final loss. Instead, plot the loss against the step number. A smooth, decreasing curve is healthy. A curve that falls painfully slowly suggests a rate that is too small. A curve that wobbles up and down means too large. A curve that shoots to infinity or turns into not a number means diverged. Always look at this curve first.

• Smooth decrease is healthy
• Flat: raise the rate
• Wobbling or NaN: lower it

So far our bowls had one bottom. A function is convex when the straight line between any two points on its graph never dips below the graph. Equivalently, it is shaped like a bowl. For a convex function, every local minimum is a global minimum, and gradient descent with a sensible learning rate will find it. Squared error with a linear model is a classic example.

• Chord stays above the graph
• One valley, one answer
• Local minimum = global minimum

Most interesting models are not convex. Neural networks give loss surfaces with many valleys and ridges. A local minimum is a point lower than all its neighbors, but not necessarily the lowest point overall. Gradient descent only sees the slope beneath its feet, so where it ends up depends on where it started. Two starting points can finish in two different valleys.

• Many valleys possible
• Local minimum is not always global
• Start point matters

There is another trap, the saddle point. Take the function x squared minus y squared. At the origin the gradient is zero, yet it is a minimum along x and a maximum along y, like the middle of a horse saddle. Gradient descent slows to a crawl near such points because the slope is almost zero. In high dimensions, saddle points are far more common than true local minima.

• Gradient zero, not a minimum
• Up in one direction, down in another
• Common in high dimensions

How do we tell these apart? The gradient being zero is only the first test. The Hessian, the matrix of second derivatives from the calculus episode, tells us the curvature. If all its eigenvalues are positive, we are at a minimum. If all are negative, a maximum. If they are mixed, positive in some directions and negative in others, it is a saddle. For x squared minus y squared the eigenvalues are two and minus two.

• Gradient zero: critical point
• Hessian eigenvalues decide
• Mixed signs: saddle

Real datasets are big. The true gradient of the cost averages the gradient over every example, so one step costs one pass over all the data. That is called batch gradient descent. With a million examples, taking a thousand steps would require a billion gradient evaluations. We need something cheaper.

• True gradient averages all examples
• One step = one full pass
• Too slow for big data

Stochastic gradient descent, or SGD, estimates the gradient from just one random example. That estimate is noisy, but on average it points in the right direction, which statisticians call unbiased. Each step is extremely cheap. The path is jittery, yet it still drifts toward the minimum, and the noise can even help shake the walker out of shallow traps.

• Gradient from one random example
• Noisy but unbiased
• Cheap steps, jittery path

The practical compromise is the mini-batch. Shuffle the data, split it into small groups such as thirty-two or sixty-four examples, and compute the gradient on each group. With one thousand examples and a batch size of thirty-two, one pass, called an epoch, contains about thirty-one updates instead of one. Larger batches give smoother gradients, and smaller batches give more updates and more noise. Hardware also runs batches efficiently in parallel.

• Batch of 32 or 64 typical
• 1000 examples: about 31 steps per epoch
• Epoch = one pass over data

Plain gradient descent has a weakness: in a narrow valley it zigzags across the steep walls and creeps along the floor. Momentum fixes this by remembering past gradients. Keep a velocity v. Each step, set v to beta times the old v plus the new gradient, then update theta by subtracting eta times v. A typical beta is zero point nine.

• Remember past gradients
• v ← βv + ∇L
• θ ← θ − ηv

Why does it work? Sideways gradients across the valley alternate in sign, so they cancel out in the running sum. Gradients along the valley floor keep the same sign, so they add up. With beta equal to zero point nine, a steady gradient can build a velocity of up to ten times the single gradient. It is like a heavy ball rolling, gathering speed in consistent directions and ignoring wiggles.

• Zigzag components cancel
• Consistent directions add up
• β = 0.9: up to 10× speed

Different parameters often need different step sizes. Adaptive optimizers track how large each parameter's gradients have been, and scale the step accordingly. Adam combines both ideas. It keeps a running average of the gradient, called m, like momentum, and a running average of the squared gradient, called v, which measures the typical size. The update divides m by the square root of v, so every parameter moves at a comparable pace.

• m: running average of gradient
• v: running average of squares
• Step = η · m / (√v + ε)

The common defaults are beta one equal to zero point nine, beta two equal to zero point nine nine nine, epsilon equal to ten to the minus eight, and learning rate zero point zero zero one. Because m and v start at zero, the early averages are biased low, so Adam divides by a correction factor. On the very first step the corrected m equals the gradient and the corrected v equals its square, so the step size is about eta, whatever the gradient scale. Adam is a great default, but tuning the learning rate still matters.

• β1 0.9, β2 0.999, ε 1e-8
• Bias correction fixes early steps
• First step size ≈ η

Time to code. We will minimize a simple two parameter function with NumPy, no libraries for optimization, and no regression datasets yet. Define f of w as the first entry squared plus ten times the second entry squared. Its gradient is a vector with two times the first entry and twenty times the second entry. Writing the gradient as a function keeps the optimizer generic.

• f(w) = w0² + 10·w1²
• grad returns a NumPy array
• Keep the optimizer generic

The core loop is only a few lines. Set w to a starting vector, say one and one, and choose eta equal to zero point zero five. Then, for each step, compute g as grad of w, and update w to w minus eta times g. Append the current loss to a list so we can plot it afterward. Notice that the update rule we derived is literally one line of code.

• Start at w = (1, 1)
• w = w − eta · g
• Record loss each step

Let us trace the first iteration by hand to verify the code. At w equals one, one, the gradient is two and twenty. Multiply by zero point zero five to get zero point one and one. Subtract, and w becomes zero point nine, zero. The second coordinate lands exactly on zero in one step, because its factor, one minus twenty times zero point zero five, is zero. The first coordinate shrinks by zero point nine five each step, so it converges more slowly.

• Gradient (2, 20)
• New w = (0.9, 0)
• x factor 0.95, y factor 0

Now experiment. Change eta to zero point zero nine and the second coordinate flips sign each step, with factor minus zero point eight, while the first creeps at zero point eight two. Change eta to zero point one one and the factor for the second coordinate is minus one point two, so it explodes and the loss grows without bound. Running these variations yourself is the best way to build intuition about the learning rate.

• η = 0.09: zigzag but converges
• η = 0.11: diverges
• Try it yourself

Adding momentum takes two more lines. Create a velocity vector of zeros. In each step compute the gradient, update v to beta times v plus g, then subtract eta times v from w. A mini-batch version looks similar, except g is computed on a random slice of the data each step. We will build exactly that when we reach linear regression, so this loop is the skeleton you will reuse for almost every model in the course.

• v = beta * v + g
• w = w − eta * v
• Same loop for every model

Let us collect the classic mistakes. Using a learning rate that is far too large or small. Forgetting the minus sign, which climbs the hill instead of descending it. Mixing up the sign of the error when deriving gradients. Using features on wildly different scales, which creates a stretched landscape just like our ten to one bowl. And judging success from one lucky run of a stochastic method instead of looking at the loss curve.

• Bad learning rate
• Wrong sign in the update
• Unscaled features
• Trusting one noisy run

A powerful debugging trick is the gradient check. Approximate the derivative numerically by evaluating the loss slightly to the right and left, subtracting, and dividing by two times the small step. Compare it with your analytic gradient. If they disagree beyond a tiny tolerance, you have a bug in your math. This uses the limit definition of the derivative from episode six, and it saves hours of confusion.

• (f(x+h) − f(x−h)) / 2h
• Compare with analytic gradient
• Mismatch means a bug

Let us recap. A loss scores one example, a cost averages the loss over the data, and training means minimizing it. MSE punishes big errors quadratically and MAE treats them linearly. Gradient descent steps against the gradient, theta minus eta times the gradient, and the learning rate decides between crawling, converging, overshooting and diverging.

• Loss, cost, objective
• MSE versus MAE
• θ ← θ − η ∇L
• Four learning-rate regimes

Convex landscapes have a single answer, while non-convex ones have local minima and saddle points. Stochastic and mini-batch methods trade accuracy of each step for speed. Momentum smooths the path, and Adam adds per-parameter scaling. And the whole thing fits in a few lines of NumPy.

• Convexity, minima, saddles
• SGD, momentum, Adam
• Next: Python and data foundations