Probability Basics and Bayes' Theorem: Why a 90% Accurate Test Can Be Wrong

A medical test is ninety percent accurate, and yours comes back positive. Are you ninety percent likely to be sick? Almost certainly not. The real answer can be closer to nine percent, and by the end of this video you will see exactly why.

• Test is 90% accurate
• You test positive
• Are you 90% likely sick?

Last time we used partial derivatives and gradients to navigate surfaces. Today we begin probability and statistics, the language machine learning uses to talk about uncertainty. Everything today builds toward one famous formula, Bayes' theorem.

• Last time: gradients
• Now: uncertainty
• Goal: Bayes' theorem

A population is every item you care about. A sample is the smaller piece you actually observe. We rarely see the whole population, so we use samples to estimate it, and a badly chosen sample gives badly biased conclusions.

• Population: everything
• Sample: what we observe
• Bias in, bias out

In a data table, each row is an observation and each column is a feature. A housing dataset might have one row per house, with columns for area, bedrooms and price. Machine learning models search for patterns across those rows.

• Row = observation
• Column = feature
• Models learn across rows

Take the numbers two, three, three, five, seven. The mean adds them up and divides by five, giving four. The median is the middle value when sorted, which is three. The mode is the most frequent value, also three.

• Data: 2, 3, 3, 5, 7
• Mean = 4
• Median = 3, mode = 3

Now replace the seven with one hundred. The mean jumps to twenty two point six, but the median stays at three. The median is robust to outliers, which is why choosing the right summary matters.

• Replace 7 with 100
• Mean jumps to 22.6
• Median stays 3

Variance measures spread. Subtract the mean from each value to get deviations of minus two, minus one, minus one, one and three. Square them to get sixteen in total, then divide by five to get three point two. The square root, about one point eight, is the standard deviation.

• Deviations from the mean
• Square, then average
• Std = square root of variance

Covariance asks whether two features move together. Multiply each feature's deviation from its mean, then average those products. Positive means they rise together. Dividing by both standard deviations gives correlation, which always lies between minus one and one.

• Do two features move together?
• Covariance: average of products
• Correlation between -1 and 1

Correlation is not causation. Ice cream sales and swimming accidents tend to rise together, but ice cream does not cause accidents. A hidden third factor, hot weather, drives both. Models find correlations, so we must be careful about what they really mean.

• Ice cream and drownings rise together
• Hidden cause: hot weather
• Correlation alone proves nothing

Probability assigns each event a number between zero and one. All possible outcomes together add up to one. So the chance that an event does not happen is one minus the chance that it does.

• Between 0 and 1
• All outcomes sum to 1
• P(not A) = 1 - P(A)

To find the chance of A or B, add their probabilities, then subtract the overlap so it is not counted twice. In a deck of fifty two cards, hearts are thirteen, kings are four, and the king of hearts is both, giving sixteen out of fifty two.

• P(A or B) = P(A)+P(B)-P(A and B)
• Avoid double counting
• Hearts or kings: 16/52

Conditional probability asks: given that B happened, how likely is A? We shrink our world to only the cases where B is true, then ask what fraction of those also contain A. That gives the probability of A and B, divided by the probability of B.

• P(A|B) = P(A and B) / P(B)
• Shrink the world to B
• Then ask how much is A

Two events are independent when knowing one tells you nothing about the other. Then the probability of both is just the product. Two fair coin flips landing heads has probability one half times one half, which is one quarter.

• P(A and B) = P(A) times P(B)
• Knowing B does not change A
• Two coin flips: 1/4

Here is the derivation. The definition of conditional probability gives P of A and B equals P of A given B times P of B. Swapping roles, it also equals P of B given A times P of A. Set them equal and divide by P of B, and you get Bayes' theorem.

• P(A and B) = P(A|B)P(B)
• P(A and B) = P(B|A)P(A)
• Set equal, divide by P(B)

Each part has a name. The prior is what you believed before seeing data. The likelihood is how probable the data is if your hypothesis is true. The evidence is the total probability of the data, and the posterior is your updated belief.

• Prior: belief before data
• Likelihood: how well data fits
• Posterior: updated belief

Suppose one percent of people have a disease. If you are sick, the test is positive ninety percent of the time. If you are healthy, it still wrongly says positive nine percent of the time. You test positive. What is the chance you are sick?

• 1% have the disease
• Sick: 90% test positive
• Healthy: 9% test positive

Imagine one thousand people. Ten are sick, and nine of them test positive. Of the nine hundred ninety healthy people, about eighty nine also test positive. So there are ninety eight positives, but only nine are truly sick.

• 10 sick, 990 healthy
• 9 true positives
• About 89 false positives

Nine out of ninety eight is about nine point two percent. Bayes' theorem agrees: zero point nine times zero point zero one is zero point zero zero nine, divided by zero point zero nine eight one in total. The disease is so rare that false alarms swamp the true cases.

• 9 / 98 is about 9.2%
• Formula gives the same result
• Rare disease dominates

We can check this with simulation. In Python, generate one million people, mark one percent as sick, and draw test results using the two accuracy rates. Among everyone who tests positive, the fraction who are truly sick settles very close to nine percent.

• Simulate one million people
• Count positives, then sick fraction
• Result converges near 9.2%

Bayesian updating means yesterday's posterior becomes today's prior. After one positive test your belief is nine point two percent. Assuming a second independent test also comes back positive, run Bayes again and the probability rises to about fifty percent.

• Posterior becomes new prior
• Second positive test
• Belief rises to about 50%

The common mistake is ignoring the base rate. This idea powers real machine learning: spam filters, Naive Bayes classifiers and many probabilistic models all update beliefs from evidence. We will build those later in the course.

• Base rate neglect
• Naive Bayes classifiers
• Probabilistic models

To recap: we summarised data with mean, median, variance and correlation, and remembered that correlation is not causation. We learned the rules of probability, derived Bayes' theorem from conditional probability, and saw that a positive test on a rare disease is far less alarming than it sounds.

• Mean, variance, correlation
• Conditional probability, independence
• Bayes: prior times likelihood

Next time we turn probability into distributions, with random variables, expected value and the central limit theorem. See you there.

• Next: random variables
• Distributions and expectation
• Central limit theorem