Distributions, Expectation and the Central Limit Theorem Explained Visually

Roll a single die and every face is equally likely, a flat boring shape. Now roll thirty dice, take the average, and repeat that thousands of times. Plot those averages and a smooth bell curve appears out of nowhere. Today you will learn exactly why that happens, and why it quietly powers a huge part of machine learning.

• Flat dice, skewed data
• Averages become bell-shaped

In the last episode we learned the rules of probability, conditional probability, and Bayes theorem. Now we upgrade from single events to whole shapes of uncertainty, called distributions. By the end you will know the five most important distributions, how to compute averages and spread, and how to reason about noisy estimates with confidence intervals and tests.

• Episode 8: probability, Bayes
• Today: distributions

Machine learning lives in noise. Labels are noisy, measurements are noisy, and the accuracy you measure on a test set is just one noisy sample of how good your model really is. A loss function is an average, and weight initialization draws random numbers from a distribution. Without this episode, you cannot tell a real improvement from luck.

• Labels are noisy
• Test accuracy is an estimate
• Losses are averages

A random variable is not random and not a variable in the usual sense. It is a function that turns each outcome of an experiment into a number. Flip a coin twice, and let X be the number of heads. The outcomes tail tail, tail head, head tail and head head map to the numbers zero, one, one and two.

• A function from outcomes to numbers
• Example: heads in two flips

The probability mass function, or PMF, lists the probability of each value. For two fair coin flips, X equals zero with probability one quarter, one with probability one half, and two with probability one quarter. Every probability is between zero and one, and they must add up to exactly one. That is the whole contract for a PMF.

• P(X = x) for each value
• Probabilities sum to 1

Some quantities, like height or time, can take any value on a continuum. Here the chance of hitting one exact value is zero, so we describe probability with a probability density function, or PDF. The probability of landing between a and b is the area under the curve between them. The total area under the whole curve is one.

• Area equals probability
• Height is density

The cumulative distribution function, or CDF, answers a simple question: what is the probability that X is at most x? For our coin example it is 0.25 at zero, 0.75 at one, and 1 at two, a staircase that only goes up. For continuous variables the CDF is the accumulated area under the PDF, so the PDF is the slope of the CDF.

• F(x) = P(X ≤ x)
• Always rises from 0 to 1

A classic mistake is believing a density value is a probability. Take a variable spread evenly between zero and one half. Its density is 2 everywhere, because the area, height times width, must equal one. A height of 2 is perfectly legal; only areas are probabilities, and those never exceed one.

• Density is not probability
• Heights can exceed 1

The simplest distribution is the Bernoulli. One trial, two outcomes: success with probability p, failure with probability one minus p. If a spam filter flags an email with probability 0.3, that is a Bernoulli with p equal to 0.3. Its mean is p, and its variance is p times one minus p, which we will prove shortly.

• One trial, success or failure
• P(1) = p

Repeat a Bernoulli trial n times independently and count the successes: that is the binomial distribution. Take four fair coin flips. The number of ways to get 0, 1, 2, 3 or 4 heads is 1, 4, 6, 4 and 1, out of sixteen equally likely sequences. So the probabilities are one sixteenth, four sixteenths, six sixteenths, four sixteenths, one sixteenth.

• Count successes in n trials
• Independent, same p

The formula is: the probability of k successes equals n choose k, times p to the k, times one minus p to the n minus k. Let n be 5, p be 0.3, and k be 2. Five choose two is 10. Point three squared is 0.09, and 0.7 cubed is 0.343. Multiply: 10 times 0.09 times 0.343 gives 0.3087, about thirty-one percent.

• P(k) = C(n,k) p^k (1−p)^(n−k)
• n = 5, p = 0.3, k = 2

Watch the binomial histogram as n grows with p fixed at 0.3. The center sits at n times p, the spread grows like the square root of n times p times one minus p, and the jagged bars slowly smooth into a bell shape. Hold on to that image, because it is the first hint of the central limit theorem.

• Mean = np
• Variance = np(1−p)
• Bars approach a bell

The Poisson distribution counts events in a fixed window when events arrive independently at an average rate lambda, like server errors per hour. Its formula is e to the minus lambda, times lambda to the k, divided by k factorial. With lambda equal to 3, the chance of exactly two events is e to the minus three, about 0.0498, times 9 over 2, which is 4.5. The result is about 0.224.

• Counts of rare events in an interval
• P(k) = e^(−λ) λ^k / k!

A Poisson is the limit of a binomial when n is huge and p is tiny while n times p stays fixed at lambda. A striking property is that its mean and its variance are both lambda. The mistake is applying it to events that cluster, like earthquake aftershocks or viral posts. If one event makes the next more likely, independence fails and the Poisson model underestimates the spread.

• Mean = variance = λ
• Limit of binomial
• Needs independent events

The uniform distribution is flat: every value between a and b is equally dense, with height one over b minus a. On the interval from zero to ten, the height is 0.1. The probability of landing between two and five is the width, 3, times 0.1, which equals 0.3. Random number generators start from this distribution, and everything else is built from it.

• Flat density 1/(b−a)
• Example: 0 to 10

The normal distribution has density one over sigma root two pi, times e to the minus x minus mu squared over two sigma squared. Mu, the mean, is where the peak sits. Sigma, the standard deviation, controls the width. Squaring the distance from the mean makes the curve symmetric, and the exponential makes the tails fall off extremely fast.

• Mean μ, standard deviation σ
• Bell curve formula

About 68 percent of a normal distribution lies within one sigma of the mean, about 95 percent within two, and about 99.7 percent within three. A z-score counts how many sigmas a value is from the mean. If exam scores have mean 70 and standard deviation 10, a score of 90 has z equal to 2, so it beats roughly 97.5 percent of people.

• ±1σ ≈ 68%, ±2σ ≈ 95%
• z = (x − μ)/σ

That z-score formula is exactly what standardization does to a feature in machine learning: subtract the mean, divide by the standard deviation. Afterward the feature has mean zero and standard deviation one. Neural network weights are also often initialized by drawing from a normal distribution with a carefully chosen small sigma.

• Subtract mean, divide by std
• Feature scaling and initialization

A common mistake is to assume all data is normal. Incomes, website traffic, and file sizes are skewed with long tails, where extreme values are far more common than a normal curve allows. Always plot a histogram before trusting a normal assumption. Later you will see that the central limit theorem rescues us in a very specific way: it applies to averages, not to the raw data.

• Income is heavily skewed
• Heavy tails break assumptions

The expected value is the probability-weighted average of all possible values: the sum of x times its probability. For a fair die, that is one plus two plus up to six, divided by six, which equals 3.5. Notice you can never actually roll 3.5. Expectation is the balance point of the distribution, the place where a seesaw would level out.

• E[X] = Σ x · P(x)
• Fair die: 3.5

A game costs 2 dollars to play. With probability 0.1 you win 10 dollars, otherwise you win nothing. The expected winnings are 10 times 0.1 plus 0 times 0.9, which is 1 dollar. Subtract the 2 dollar cost and the expected profit is minus 1 dollar per play. Any single game can go well, but on average you lose.

• Ticket costs 2
• Win 10 with probability 0.1

Expectation is linear. Scaling a variable scales its mean, shifting it shifts the mean, and the expectation of a sum is the sum of expectations, even when the variables are dependent. Two dice add up to 3.5 plus 3.5, which is 7, with no need to list all 36 outcomes. This one property makes many hard problems easy.

• E[aX + b] = aE[X] + b
• E[X + Y] = E[X] + E[Y]

Mean alone hides spread. Variance is the expected squared distance from the mean: the average of x minus mu, squared. We square so that positive and negative deviations cannot cancel, and so large deviations count extra. Standard deviation is the square root of variance, which returns the units to those of the data.

• Var(X) = E[(X − μ)²]
• Std = √Var

Start with the variance, the expectation of x minus mu, squared. Expand the square: x squared minus two mu x plus mu squared. By linearity, that is E of x squared, minus two mu times E of x, plus mu squared. Since E of x is mu, the middle term is minus two mu squared. Combine with the last term, and you get E of x squared minus mu squared.

• Expand (X − μ)²
• Use linearity
• Var = E[X²] − E[X]²

Check it on a die. The squares one, four, nine, sixteen, twenty-five and thirty-six sum to 91, so E of x squared is 91 over 6, about 15.167. The mean squared is 3.5 squared, which is 12.25. The difference is about 2.917, so the standard deviation is about 1.708. Computing the variance directly from deviations gives the same number.

• E[X²] = 91/6 ≈ 15.167
• E[X]² = 12.25
• Var ≈ 2.917

Shifting a variable does not change its variance, but scaling by a multiplies the variance by a squared. For independent variables, variances add, so the variance of a sum is the sum of the variances. Two classic mistakes: thinking the variance of a difference subtracts, when it also adds, and thinking E of x squared equals E of x, squared. They differ by exactly the variance.

• Var(aX + b) = a²Var(X)
• Independent: variances add
• Var(X − Y) also adds

Models never see one variable alone. A joint distribution gives the probability of combinations. Let X be weather, rain or sun, and Y be whether you are late. Suppose rain and late has probability 0.2, rain and on time 0.1, sun and late 0.1, and sun and on time 0.6. These four numbers sum to one, and they define the whole world.

• Two variables at once
• Table of P(X, Y)

To recover the distribution of one variable alone, sum across the other. This is called marginalizing. The probability of rain is 0.2 plus 0.1, which is 0.3, and sun is 0.7. The probability of being late is 0.2 plus 0.1, also 0.3. The totals sit in the margins of the table, which is where the name comes from.

• Sum out the other variable
• P(rain) = 0.3, P(late) = 0.3

A conditional distribution zooms into one row and renormalizes. Given rain, the chance of being late is 0.2 divided by 0.3, about 0.667. Given sun, it is 0.1 divided by 0.7, about 0.143. If weather and lateness were independent, the joint for rain and late would be 0.3 times 0.3, which is 0.09, not 0.2. So they are clearly dependent.

• P(Y|X) = P(X,Y)/P(X)
• Independent only if joint = product

Here is the big connection. A supervised dataset is a collection of samples drawn from an unknown joint distribution of features X and labels Y. A classifier tries to estimate the conditional distribution of Y given X. Everything about generalization, which we study later, is about whether what we learned from samples holds for the whole distribution.

• Data = samples from P(X, Y)
• Classifier learns P(Y | X)

Now to the first great theorem. The law of large numbers says that as you collect more independent samples, the sample average converges to the true expected value. Simulate rolling a die: after ten rolls the average might be 3.1 or 4.2, but after ten thousand rolls it hugs 3.5 tightly. This is why averaging over more data works.

• Sample mean approaches E[X]
• More data, less error

How fast does the average settle? For a fair coin, the proportion of heads has standard deviation 0.5 divided by the square root of n. With 10 flips that is about 0.16, so 70 percent heads is not shocking. With 10,000 flips it is 0.005, so you will almost always land between 49 and 51 percent. The error shrinks like one over the square root of n.

• Std of proportion = 0.5/√n
• n = 10000: about 0.005

The law does not say that a coin owes you tails after a run of heads. Each flip is independent; the past does not compensate. The proportion converges only because new flips dilute old ones. In fact, the gap between the number of heads and tails typically grows, on the order of the square root of n, even as the proportion approaches one half.

• Coins have no memory
• Averages converge, counts do not balance

The law of large numbers tells us where averages land. The central limit theorem tells us the shape of how they scatter. Take a very skewed distribution, say waiting times that are mostly short with a long tail. Draw 30 samples, average them, record the result, and repeat 10,000 times. Then histogram those averages.

• Take n samples, average them
• Repeat many times

The result is a bell curve, even though the original data was skewed. Formally, if you average n independent samples with mean mu and standard deviation sigma, the average is approximately normal, centered at mu, with standard deviation sigma over the square root of n. That quantity is called the standard error. Notice the square root of n again.

• Mean ≈ μ
• Std error = σ/√n
• Shape ≈ normal

Average 30 die rolls. The mean is 3.5 and sigma is about 1.708, so the standard error is 1.708 divided by the square root of 30, about 0.312. What is the chance the average exceeds 4? The z-score is 4 minus 3.5, divided by 0.312, about 1.60. The upper tail beyond 1.6 is roughly 0.055, so about one in eighteen.

• Die: σ ≈ 1.708, n = 30
• SE ≈ 0.312
• P(mean > 4) ≈ 0.055

Intuitively, an average mixes many small independent pushes, some up and some down. Extreme outcomes need many pushes to line up in one direction, which is rare, while middling outcomes can happen in countless ways. Adding independent variables repeatedly smooths out whatever quirks the original shape had, and the bell curve is what remains. More skewed data just needs a larger n.

• Many small independent pieces
• Extremes cancel out

The theorem has fine print. The variance must be finite, so heavy-tailed distributions like the Cauchy have averages that never settle into a bell curve. Samples must be independent, or at least weakly dependent, so correlated time series can break it. And n needs to be large enough for the data at hand. Skipping these checks is a common mistake.

• Needs finite variance
• Needs near-independence
• Cauchy: no convergence

The theorem is everywhere in machine learning. Test accuracy is an average of correct or incorrect indicators, so it is approximately normal with a standard error we can compute. A mini-batch gradient is an average over a few examples, so it is a noisy estimate of the true gradient. And because error shrinks like one over root n, quadrupling your data only halves the noise.

• Accuracy is an average
• Mini-batch gradients are noisy averages
• Error shrinks as 1/√n

We rarely know the true mean, only a sample. The CLT lets us build an interval around our estimate. A 95 percent confidence interval is the sample mean plus or minus 1.96 standard errors. Say 100 samples have mean 82 and standard deviation 10. The standard error is 10 over 10, which is 1, so the interval is roughly 80.04 to 83.96.

• Estimate ± 1.96 × SE
• Covers 95%

Here is the subtle part. The true mean is a fixed number, not random. What is random is the interval, which changes with every sample. The 95 percent refers to the procedure: if you repeated the study many times, about 95 percent of the intervals would contain the truth. Simulate 100 intervals, and roughly five will miss it.

• The method covers truth 95% of the time
• Not: 95% chance this interval holds it

Your model scores 90 percent on 400 test examples. For a proportion, the standard error is the square root of p times one minus p, over n, which is the square root of 0.09 over 400, equal to 0.015. Multiply by 1.96 to get 0.029. The interval is about 87.1 to 92.9 percent. A rival model scoring 91 percent is well inside that noise.

• 90% on 400 test examples
• SE = 0.015
• Interval ≈ 87.1% to 92.9%

Hypothesis testing asks whether data is surprising under a boring assumption, the null hypothesis. Suppose we flip a coin 100 times and get 60 heads. Under the null that the coin is fair, the mean is 50 and the standard deviation is the square root of 100 times 0.25, which is 5. The z-statistic is 60 minus 50, over 5, equal to 2.

• Null hypothesis H0
• Test statistic
• How surprising is the data?

The p-value is the probability, assuming the null is true, of seeing a result at least as extreme as ours. A z of 2, counting both tails, gives roughly 0.046. That is under the usual cutoff of 0.05, so we say the result is statistically significant. Note what it is not: it is not the probability that the coin is fair.

• P(data this extreme | H0)
• Two-sided z = 2 gives p ≈ 0.046

Pitfall one: running many tests. If you try 20 unrelated features, each at the 0.05 level and with no real effects, the chance of at least one false alarm is one minus 0.95 to the 20, about 0.64. Trying many things and reporting only what worked is called p-hacking. In machine learning, comparing dozens of hyperparameter settings on one test set does exactly this.

• 20 tests at 0.05
• 1 − 0.95^20 ≈ 0.64

Pitfall two: confusing significance with importance. With millions of samples, even a difference of 0.01 percent in accuracy can have a tiny p-value, yet be worthless in practice. And a large, useful effect in a small sample can fail to reach significance. Always report the effect size with its confidence interval, and decide in advance how many samples you will collect, rather than stopping when the p-value looks good.

• Huge n finds tiny effects
• Report effect size and intervals

Let us recap. A random variable maps outcomes to numbers, described by a PMF or PDF and a CDF. Bernoulli, binomial, Poisson, uniform and normal cover most situations. The mean is a weighted average, and variance equals E of x squared minus E of x, squared. Averages converge by the law of large numbers, become bell shaped by the CLT, and that gives us intervals and tests, with their pitfalls.

• Distributions, PMF, PDF, CDF
• Expectation and variance
• LLN and CLT
• Intervals and tests

We now know what distributions look like, but how do we choose the right one to fit our data? That is the subject of the next episode: maximum likelihood, entropy, and cross-entropy, the idea behind most ML loss functions. See you there.

• Fit distributions to data
• Entropy and cross-entropy
• Subscribe for episode 10