Confidence Intervals and Hypothesis Tests for ML: Is Your Model Really Better?

Your new model scores ninety percent on the test set. The old one scored eighty-eight. Do you ship it? Before you answer, ask a harder question: if you collected a different test set, would the new model still win? This lesson gives you the tools to answer that honestly.

• Old model: 88%
• New model: 90%
• Real gain or luck?

In lesson thirty-eight we studied covariance, correlation and the multivariate Gaussian. Today we move from describing data to reasoning under uncertainty. We will study sampling distributions, estimators, confidence intervals, bootstrapping, hypothesis tests, and the traps of multiple testing. We build on the central limit theorem from lesson nine.

• Sampling and estimators
• Confidence intervals
• Hypothesis tests
• Error bars on accuracy

A population is everything we care about, such as every future user your model will see. A sample is the small slice we actually observe, like a test set. A parameter, such as the true accuracy, is a fixed unknown number. A statistic, such as test accuracy, is computed from the sample and changes every time we draw a new sample.

• Population: everything
• Sample: what we observe
• Statistic estimates parameter

Imagine drawing a sample of twenty-five, computing its mean, writing it down, then repeating thousands of times. Those means are not all identical. Plotted, they form a histogram called the sampling distribution of the mean. It describes how much a statistic wobbles from sample to sample, and it is the foundation of everything today.

• Draw a sample, compute the mean
• Repeat many times
• Means form their own distribution

The central limit theorem says the sample mean is roughly normal, centered on the true mean, with standard deviation sigma divided by the square root of n. We call this the standard error. Be careful: standard deviation describes how spread out individual data points are, while standard error describes how spread out our estimate is.

• SE = sigma / sqrt(n)
• Spread of the sample mean
• Not the spread of the data

Suppose individual values have standard deviation ten. With twenty-five samples, the standard error is ten divided by five, which is two. With one hundred samples it is ten divided by ten, which is one. Quadrupling the data only halves the error. That square root is why gathering more data has diminishing returns.

• sigma = 10, n = 25: SE = 2
• n = 100: SE = 1
• 4 times data, half the error

An estimator is a rule that turns data into a guess for a parameter. The sample mean is an estimator of the population mean. Because the data are random, the estimator is random too. So we judge an estimator by looking at its sampling distribution: where is it centered, and how wide is it?

• A rule that turns data into a guess
• Itself a random variable
• Judge it by its distribution

Bias is the gap between the average value of the estimator and the true parameter. An unbiased estimator is right on average, though any single estimate can still miss. The sample mean is unbiased. Think of two archers: one scatters evenly around the bullseye, the other clusters tightly but off to the side. Both have a problem, of different kinds.

• Bias = E[estimate] - truth
• Unbiased: right on average
• Low bias is not low error

An estimator is consistent if it homes in on the truth as the sample grows. The sample mean is consistent, thanks to the law of large numbers. Notice that consistency and unbiasedness are different ideas. An estimator can be biased for small n but still consistent, because its bias fades away as data accumulates.

• More data, estimate converges
• Law of large numbers
• Spread shrinks as n grows

Here is the famous puzzle. The sample variance divides by n minus one, not n. The reason is that the sample mean is fitted to this very sample, so the points sit closer to it than to the true mean. Squared deviations around the sample mean come out too small on average, and dividing by n minus one corrects exactly that shortfall.

• Data sits close to its own mean
• Deviations from x-bar are too small
• Divide by n-1 to correct

Take the three values two, four and nine. The mean is five. The deviations are minus three, minus one and plus four, and their squares sum to twenty-six. Dividing by n gives eight point six seven. Dividing by n minus one gives thirteen. For small samples the difference is large, which is why it matters for tiny test sets.

• Data: 2, 4, 9, mean 5
• Squares: 9 + 1 + 16 = 26
• Divide by 3: 8.67; by 2: 13

Another view is degrees of freedom. Once the mean is fixed, only n minus one deviations can vary freely, because the last one must make them sum to zero. A common mistake in Python is that NumPy's var and std default to dividing by n. Pass ddof equals one to get the sample version. Pandas already uses n minus one by default.

• n values, one mean fitted
• Only n-1 are free
• np.var(x, ddof=1)

A single number like eighty is a point estimate, and it hides how uncertain we are. A confidence interval reports a range instead. The recipe is simple: take the estimate, then add and subtract a margin of error. The margin is a critical value times the standard error, so both our uncertainty and our confidence level shape the width.

• Point estimate alone hides uncertainty
• Interval = estimate ± margin
• Margin = critical value × SE

Say the sample mean is eighty, the known standard deviation is twelve, and n is thirty-six. The standard error is twelve over six, which is two. For ninety-five percent confidence the critical value from the normal curve is one point nine six. The margin is three point nine two, so the interval runs from seventy-six point oh eight to eighty-three point nine two.

• Mean 80, sigma 12, n = 36
• SE = 12/6 = 2
• 80 ± 1.96×2 = [76.08, 83.92]

In practice we never know sigma, so we plug in the sample standard deviation s. That adds extra uncertainty, especially for small samples, so the right curve is the Student t distribution. It looks like a normal curve with heavier tails, and its shape depends on degrees of freedom, n minus one. As n grows, it merges with the normal.

• Replace sigma with s
• Extra uncertainty, fatter tails
• Use t with n-1 degrees of freedom

Suppose ten validation scores have mean fifty and sample standard deviation six. The standard error is six over root ten, about one point eight nine seven. With nine degrees of freedom the ninety-five percent critical value is two point two six two, wider than one point nine six. The margin is about four point two nine, giving roughly forty-five point seven to fifty-four point three.

• n = 10, mean 50, s = 6
• SE = 6/√10 ≈ 1.897
• t* = 2.262, margin ≈ 4.29

Now the most misunderstood idea in statistics. A ninety-five percent interval does not mean there is a ninety-five percent chance the true value lies inside this particular interval. The true value is fixed. What is random is the interval. If we repeated the whole procedure many times, about ninety-five percent of the intervals would capture the truth.

• Not: 95% chance mu is inside
• The method captures mu 95% of the time
• The truth is fixed; the interval moves

Three dials control the width. Demanding more confidence, say ninety-nine percent instead of ninety-five, widens the interval. Collecting more data narrows it, at the rate of one over root n. And noisier data, with larger s, widens it. There is no free lunch: you cannot be both very certain and very precise without more data.

• Higher confidence, wider interval
• More data, narrower interval
• Noisier data, wider interval

The z and t formulas rely on assumptions, and some statistics, like the median or an F1 score, have no easy formula. The bootstrap is a clever workaround. Treat your sample as a stand-in for the population. Draw new samples of the same size from it with replacement, compute your statistic on each, and repeat thousands of times.

• Treat the sample as the population
• Resample with replacement
• Compute the statistic each time

After ten thousand resamples you have ten thousand values of the statistic. Sort them and read off the two point five and ninety-seven point five percentiles. That range is your ninety-five percent bootstrap interval. No normality assumption is needed, and it works for almost any statistic, though it is unreliable for very tiny samples.

• Collect 10,000 bootstrap statistics
• Sort them
• Take the 2.5th and 97.5th percentiles

In Python this takes a few lines. Create a random generator, then in a loop call rng dot choice on your data with replace set to true, compute the statistic, and store it. Finally call numpy percentile with two point five and ninety-seven point five. For a model, resample the test examples, recompute accuracy each time, and you have an interval.

• rng.choice(x, size=n, replace=True)
• Repeat 10,000 times
• np.percentile(stats, [2.5, 97.5])

Hypothesis testing flips the question. We start with a null hypothesis, usually the boring claim that nothing is going on, such as two models having equal true accuracy. The alternative says there is a difference. Then we ask: if the null were true, how surprising would our data be? Surprising data casts doubt on the null.

• H0: no effect, the boring story
• H1: there is an effect
• Ask: how surprising is the data under H0?

The p-value is the probability, assuming the null is true, of seeing a result at least as extreme as ours. A small p-value means our data would be rare if nothing were happening. Common mistake: the p-value is not the probability that the null is true, and it says nothing about how large or important an effect is.

• P(data this extreme | H0 true)
• Small p: data is surprising under H0
• NOT the probability H0 is true

Tests make two kinds of mistakes. A type one error rejects a true null, a false alarm, and we cap its rate with the significance level alpha, often zero point oh five. A type two error misses a real effect, with probability beta. Power is one minus beta: the chance of detecting a real effect. Low power means real improvements often go unnoticed.

• Type I: false alarm, probability alpha
• Type II: missed effect, probability beta
• Power = 1 - beta

The one-sample t-test measures how many standard errors the sample mean sits from the null value. With mean fifty, null value forty-seven, s equal six and n equal ten, the standard error is one point eight nine seven, so t is about one point five eight. With nine degrees of freedom the two-sided p-value is about zero point one five. Not enough to reject.

• t = (x-bar - mu0) / (s/√n)
• Example: 50 vs 47, t ≈ 1.58
• df = 9, p ≈ 0.15

Now a realistic case. Model A answers four hundred sixty of five hundred requests correctly, ninety-two percent. Model B gets four hundred forty, eighty-eight percent, on a different group of five hundred. A four-point gap looks decent. The null says both models have the same true accuracy. Could this gap be luck?

• Model A: 460 of 500 correct (92%)
• Model B: 440 of 500 correct (88%)
• H0: same true accuracy

Under the null, pool the data: nine hundred correct out of one thousand, so p is zero point nine. The standard error of the difference is the square root of zero point nine times zero point one times two over five hundred, about zero point oh one nine. The gap of zero point oh four gives z of about two point one one, and a two-sided p-value near zero point oh three five.

• Pooled p = 900/1000 = 0.9
• SE = √(0.9×0.1×2/500) ≈ 0.019
• z ≈ 2.11, p ≈ 0.035

The chi-square test reaches the same answer from a table of counts. If both models were equal, each would be expected to get four hundred fifty right and fifty wrong. Add up observed minus expected, squared, divided by expected, for all four cells: ten squared over four fifty twice, plus ten squared over fifty twice. That is zero point four four plus four, so four point four four.

• Table: correct / wrong × A / B
• Expected: 450 correct, 50 wrong each
• χ² = Σ (O−E)²/E ≈ 4.44

With one degree of freedom the cutoff is three point eight four, and four point four four exceeds it, so p is about zero point oh three five. Notice that z squared is the chi-square statistic, the same test in disguise. But also notice a warning: if the true gap really were four points, this sample size would detect it only about fifty-six percent of the time. Power matters.

• 4.44 > 3.84, so p ≈ 0.035
• z squared equals chi-square
• Power here only about 56%

Suppose you try twenty ideas, each tested at the five percent level, and none truly works. The chance that all twenty correctly stay quiet is zero point nine five to the twentieth power, about thirty-six percent. So there is roughly a sixty-four percent chance of at least one false discovery. Test enough things and something will look significant by luck.

• 20 tests at alpha = 0.05
• P(at least one false alarm) = 1 − 0.95²⁰
• ≈ 64%

The simplest fix is the Bonferroni correction: divide alpha by the number of tests. With twenty tests, each must reach p below zero point zero zero two five. This keeps the overall false-alarm chance near five percent. It is conservative, so it costs some power, and there are gentler methods, but the lesson is to account for how many things you tried.

• Divide alpha by number of tests
• 20 tests: 0.05/20 = 0.0025
• Conservative but simple

P-hacking is, often unintentionally, searching until something is significant and reporting only that. In machine learning it looks like trying many random seeds, features and hyperparameters, then reporting the best test score. Every look at the test set is another test. Decide your comparison before running experiments, record everything you tried, and keep a truly untouched final test set.

• Try many seeds, features, splits
• Report only the winner
• Tuning on the test set is p-hacking

Test accuracy is a sample mean of ones and zeros, so it has a standard error too. For accuracy p on n examples, the standard error is the square root of p times one minus p over n. With ninety percent on one thousand examples, that is about zero point zero zero nine five. The ninety-five percent interval is roughly plus or minus 1.9 points: eighty-eight point one to ninety-one point nine.

• Each test example: correct or not
• SE = √(p(1−p)/n)
• p = 0.90, n = 1000: SE ≈ 0.0095

Shrink the test set to one hundred examples and the standard error triples to zero point oh three, so the interval is about plus or minus six points. That means a one-point leaderboard gain on a small test set is mostly noise. Always report an interval, using the formula or a bootstrap over test examples, and for model comparisons use a paired test on the same examples.

• n = 100: SE = 0.03
• Interval ≈ ±5.9 points
• 1-point gains are often noise

Let us recap. A statistic wobbles, and its sampling distribution has standard error sigma over root n. Sample variance divides by n minus one to stay unbiased. Confidence intervals come from z, t or the bootstrap, and mean something about the procedure, not one interval. Tests use p-values, with type one and two errors and power. Beware multiple comparisons, and always put error bars on accuracy.

• SE = sigma/√n; divide variance by n−1
• CI: estimate ± critical × SE
• p-value, alpha, power; correct for many tests

Next, in lesson forty, we use these ideas to build linear regression from scratch in NumPy, checking its assumptions and analyzing residuals.

• Lesson 40: NumPy implementation
• Assumptions and residual analysis
• Subscribe for the full course