Maximum Likelihood, Entropy and Cross-Entropy Explained From Scratch

Almost every classifier you will ever train, from a simple logistic regression to a giant language model, minimizes the same formula: cross-entropy. Most courses hand it to you and move on. Today we derive it from nothing, using only coins, a few bell curves, and a calculator.

• Classifiers minimize cross-entropy
• Why that formula?

Last time we met distributions, expectation, and the central limit theorem. Now we flip the question around. Instead of asking what data a distribution produces, we ask which distribution most likely produced the data we actually have.

• Episode 9: distributions
• Expectation and the CLT

Here is the plan. First, likelihood and the maximum likelihood estimate, derived for a coin and for a Gaussian. Then the log trick and why computers need it. Then entropy, cross-entropy, and KL divergence. And finally the punchline: minimizing cross-entropy and maximizing likelihood are the exact same thing.

• Likelihood and MLE
• Entropy and KL divergence
• Cross-entropy equals MLE

Probability looks forward. Suppose we know a coin has a probability of heads equal to one half. Then we can ask how likely seven heads in ten flips would be. The parameter is fixed, the data varies, and the probabilities over all possible outcomes add up to one.

• Parameter fixed, data varies
• Sums to one over outcomes

Likelihood looks backward. Now the data is fixed: we saw seven heads and three tails. We treat the same formula as a function of the unknown parameter p. We write L of p, and it tells us how well each candidate value of p explains what we saw. It does not need to add up to one.

• Data fixed, parameter varies
• Does not sum to one

For independent flips, probabilities multiply. Each head contributes a factor of p, and each tail contributes a factor of one minus p. With seven heads and three tails, the likelihood is p to the seventh power times one minus p to the third power. The order of flips does not matter for finding the best p.

• Independent flips multiply
• L(p) = p^7 (1-p)^3

Let us just try some values. At p equals point three, the likelihood is about zero point zero zero zero zero seven five. At point five it is about zero point zero zero zero nine seven seven. At point seven it is about zero point zero zero two two two four, more than twice as high as at one half. The data prefers point seven.

• p=0.3: 0.000075
• p=0.5: 0.000977
• p=0.7: 0.002224

That is the whole idea of maximum likelihood estimation. Among all possible parameter values, choose the one that makes the observed data most probable. We write it as theta hat equals the argmax of L of theta. Next we find the peak exactly, using calculus from episode six.

• Pick parameters that make data likeliest
• theta-hat = argmax L(theta)

We could differentiate p to the seventh times one minus p cubed using the product rule, but it gets ugly fast. There is a cleaner way. The logarithm is increasing, so the peak of the likelihood sits at the same p as the peak of its log. And the log turns products into sums.

• Products are messy
• Log turns products into sums

Taking logs gives the log-likelihood: seven times the natural log of p, plus three times the natural log of one minus p. Notice the peak is in the same place. Both curves top out at point seven, but the bottom curve is far easier to differentiate.

• l(p) = 7 ln p + 3 ln(1-p)
• Same peak, easier algebra

Differentiate: the derivative of seven log p is seven over p, and the derivative of three log of one minus p is minus three over one minus p. Set the sum to zero. Multiply through: seven times one minus p equals three p. That is seven minus seven p equals three p, so seven equals ten p, and p equals point seven.

• 7/p - 3/(1-p) = 0
• 7(1-p) = 3p
• p = 7/10

Redo it with letters. With k heads in n flips, the derivative is k over p minus n minus k over one minus p. Setting it to zero gives p hat equals k over n. The maximum likelihood estimate is simply the observed fraction of heads. Math just confirmed the obvious guess, which is a good sign.

• k heads in n flips
• p-hat = k / n

A zero derivative could be a peak or a valley, so check the curvature. The second derivative is minus k over p squared minus n minus k over one minus p squared, which is always negative. The curve is concave, so our stationary point is a true maximum.

• Concave: a true maximum
• Not a minimum

Now continuous data. Say we measured three values: two, four, and six, and we believe they come from a Gaussian with unknown mean mu and known standard deviation one. For each point, the Gaussian density gives its height. The likelihood is the product of those three heights.

• Density replaces probability
• Data: 2, 4, 6

Picture sliding the bell curve left and right. When it sits at three, the point six is far in the tail and gets a tiny height. When it sits at four, the heights balance, and the product is largest. Our eyes say four. Now let us derive it.

• mu=3 fits poorly
• mu=4 fits best

The Gaussian density has a constant out front and an exponential of minus x minus mu, squared, over two sigma squared. Taking the log of a product of densities gives a sum, and the exponential disappears. We get minus the sum of squared deviations over two sigma squared, plus a constant that does not depend on mu.

• Density: exp(-(x-mu)^2 / 2 sigma^2)
• Log removes the exp

Differentiate with respect to mu. The result is the sum of x i minus mu, divided by sigma squared. Set it to zero, so the sum of x i equals n times mu. Therefore mu hat is the sample average. For two, four, and six, that is twelve over three, which is four.

• Derivative: sum (x_i - mu) / sigma^2
• Set to zero: mu = average

Look at what happened. Maximizing the Gaussian log-likelihood is the same as minimizing the sum of squared errors. At mu equal three, the squared errors are one, one, and nine, totaling eleven. At four they are four, zero, and four, totaling eight. This is why mean squared error is the natural loss under Gaussian noise.

• Maximize likelihood = minimize squared error
• mu=4: SSE 8; mu=3: SSE 11

A common mistake: assuming MLE always gives the best estimator. If we also estimate the variance, the maximum likelihood answer divides by n, giving eight over three, about two point six seven, for our data. The unbiased version divides by n minus one and gives four. MLE is consistent, but can be slightly biased in small samples.

• MLE variance divides by n
• Unbiased divides by n-1

Real datasets have thousands or millions of points. Each likelihood factor is a probability below one, so the product shrinks relentlessly. With one thousand flips of a fair coin, the likelihood is zero point five to the one thousand, around nine times ten to the minus three hundred and two. Still representable, but just barely.

• 1000 data points
• Each probability below 1

A standard double-precision float cannot go much below about ten to the minus three hundred and eight, with denormal numbers stretching a little further. Push past that and the result silently becomes exactly zero. Multiply two thousand probabilities of one half and Python happily reports a likelihood of zero, for every candidate parameter. Your optimizer then sees a flat, useless landscape.

• Smallest normal double ~ 2e-308
• 0.5^2000 becomes 0.0

The fix is to work with logs. The log of the product is the sum of the logs. Each factor contributes minus zero point six nine three, and two thousand of them total minus one thousand three hundred eighty six point two nine. That is a perfectly ordinary number. No underflow, and sums are also kinder to rounding errors than long products.

• Sum of logs, not product
• 2000 x ln 0.5 = -1386.29

By convention, optimizers minimize things. So we flip the sign and minimize the negative log-likelihood, written NLL. Same answer, nicer packaging. Remember this name. You will see it again, disguised as cross-entropy, in about eight minutes.

• Optimizers minimize
• NLL = - sum log p(x_i)

Two more habits. Never take the log of exactly zero, because it gives minus infinity, so clip probabilities to a tiny epsilon. And when summing exponentials, as in softmax, subtract the maximum first. That is the log-sum-exp trick, and it stops large exponentials from overflowing.

• Never log zero: add epsilon or clip
• Use log-sum-exp for softmax

MLE has a weakness. Flip a coin three times and see three heads. The maximum likelihood estimate is k over n, which is exactly one. The model now claims tails are impossible. With so little data, that is overconfident, and common sense disagrees.

• 3 flips, 3 heads
• MLE says p = 1.0

Bayes' theorem from episode eight offers a fix. Encode a prior belief about p, for instance that fair coins are common. Then multiply prior by likelihood to get a posterior. The posterior is proportional to likelihood times prior, and its peak balances data against belief.

• Prior: p is probably near 0.5
• Posterior ∝ likelihood × prior

Maximum a posteriori estimation, or MAP, maximizes log likelihood plus log prior. With a mild Beta two-two prior, the estimate becomes k plus one over n plus two. Three heads in three flips now gives four fifths, point eight, not one. Seven heads in ten gives eight twelfths, about point six six seven.

• Beta(2,2) prior
• MAP = (k+1)/(n+2)

As data grows, the prior fades and MAP converges to MLE. Here is a preview of episode seventeen: a Gaussian prior on model weights turns into an L2 penalty added to the loss. Regularization is just a prior in disguise. For now, treat MLE as the data-only view and MAP as MLE plus common sense.

• More data, prior fades
• Gaussian prior gives L2 penalty

Now information theory. How surprised should you be by an event? A certain event should surprise you zero. A rare event should surprise you a lot. The measure that fits is minus log of p. In base two, an event with probability point nine carries about point one five two bits, while probability point one carries about three point three two bits.

• Rare events surprise more
• Surprise = -log2 p

Why a logarithm? Because independent events should have additive surprise. Two fair dice landing on particular faces has probability one thirty-sixth. The surprise is the sum of the two individual surprises, about two point five eight five bits each, for five point one seven total. Only the logarithm turns multiplication into addition.

• Independent events add surprise
• p1 x p2 becomes sum

Entropy is the average surprise, weighted by how often each outcome happens. The formula: H of p equals minus the sum over outcomes of p i times log p i. Here p i is the probability of outcome i. It is the expectation from episode nine, applied to surprise.

• H(p) = - sum p log p
• Expected surprise

Compute it. A fair coin: half times one bit, plus half times one bit, equals one bit. A coin that lands heads ninety percent of the time: point nine times point one five two is about point one three seven, plus point one times three point three two two is about point three three two. Total: about point four six nine bits. Predictable means low entropy.

• Fair coin: 1 bit
• Biased 0.9/0.1: 0.469 bits

For a coin, entropy as a function of p is an arch. It is zero at p equals zero or one, where nothing is uncertain, and peaks at one bit at p equals one half. Entropy is also the minimum average number of bits per symbol needed to encode outcomes from that distribution. That compression view matters next.

• Max when all outcomes equal
• Zero when certain

Suppose reality follows distribution p, but your model believes distribution q. You build a code, or a set of predictions, tuned for q, yet the events keep arriving from p. How many bits do you now pay on average? That quantity is the cross-entropy.

• True distribution p
• Model belief q

The formula changes just one letter. Cross-entropy of p and q is minus the sum of p i times log q i. The weights come from reality, p, but the surprise is measured by the model, q. When q equals p, it collapses to ordinary entropy.

• H(p,q) = - sum p log q
• Events from p, surprise from q

Let the truth be point nine, point one. A model that says fifty-fifty pays one bit on average. A better model, point eight and point two, pays point nine times point three two two, about point two nine, plus point one times two point three two two, about point two three two. That totals about point five two two bits. Closer to the truth costs less.

• q=(0.5,0.5): 1.0 bits
• q=(0.8,0.2): 0.522 bits

The extra cost beyond the unavoidable entropy is the KL divergence. Cross-entropy minus entropy equals the sum of p i times log of p i over q i. For q equal point eight, point two, that is point five two two minus point four six nine, about point zero five three bits. For the fifty-fifty model, it is point five three one bits.

• KL(p||q) = H(p,q) - H(p)
• Extra bits from wrong model

KL divergence is always greater than or equal to zero, and equals zero only when q matches p. The proof uses the fact that log x is at most x minus one. One warning: KL is not symmetric, so KL of p from q differs from KL of q from p. That is why we do not call it a distance.

• KL >= 0, zero only if p = q
• Not symmetric: not a true distance

Now connect the pieces. In classification, each training example has a label. As a distribution, that label is one-hot: probability one on the correct class, zero elsewhere. The model outputs its own distribution q over classes. Cross-entropy compares the two for each example.

• One-hot labels
• Empirical distribution of data

Plug in a one-hot p. Every term in the sum is multiplied by zero except the true class. So the cross-entropy for that example is just minus the log of the probability the model assigned to the correct answer. If the model gave point eight, the loss is minus natural log of point eight, about point two two three.

• Only true class term survives
• CE = - log q(true class)

The likelihood of the whole dataset is the product, over examples, of the probability the model gives the correct label. Take the log, negate it, and divide by n. You get the average of minus log q of the true class. That is exactly the average cross-entropy loss. Different names, identical objective.

• Likelihood = product of q(y_i)
• Log, negate, divide by n

A tiny dataset. The model gives the correct class probabilities point eight, point six, and point nine. The likelihood is their product, point four three two. The negative log of that is point eight three nine. Alternatively, average the three individual losses: point two two three, point five one one, and point one zero five, for a mean of about point two eight.

• Probabilities: 0.8, 0.6, 0.9
• Product 0.432, NLL 0.839

So training a classifier with cross-entropy is maximum likelihood estimation. Since entropy of the data does not depend on the model, minimizing cross-entropy is also minimizing the KL divergence from the data distribution to the model. Three viewpoints, one optimization. This is why the loss is not arbitrary.

• Minimizing CE = maximizing likelihood
• Same for KL to the data

In Python, build a grid of p values, compute seven times np dot log of p plus three times np dot log of one minus p, and take the argmax: you get point seven. Then print point five to the power two thousand and see zero. Print two thousand times np dot log of point five and see minus one thousand three hundred eighty six point two nine.

• np.log for log-likelihood
• 0.5**2000 equals 0.0

Entropy is minus np dot sum of p times log base two of p, giving point four six nine. Cross-entropy swaps in log of q and gives point five two two. Subtract and you get KL, point zero five three. Your hand calculations match the machine, which is the verification step we always want.

• H = -sum(p * log2(p))
• KL = sum(p * log2(p/q))

Common mistakes. Reading likelihood as the probability that a parameter is true. Taking the log of zero. Mixing bits, which use log base two, with nats, which use the natural log, since libraries like PyTorch use natural logs. And treating KL as symmetric. Each one has bitten real projects.

• Confusing likelihood with probability
• Forgetting log of zero
• Mixing bits and nats

Recap. Likelihood scores parameters given fixed data. The coin MLE is k over n, and the Gaussian mean MLE is the sample average. Logs turn products into sums and save us from underflow. Entropy is average surprise, KL is the extra cost of a wrong model, and minimizing cross-entropy equals maximizing likelihood.

• MLE: best parameters for data
• Log-likelihood stops underflow
• CE = NLL = MLE

Next episode, we take these losses, mean squared error and cross-entropy, and learn how to actually minimize them with gradient descent, learning rates, and optimizers. See you in episode eleven.

• Episode 11: Loss functions and gradient descent
• Subscribe for the full series