Bayesian ML: MAP, Priors, and Gaussian Processes Explained

Imagine two weather apps. One says tomorrow will be twenty degrees. The other says twenty degrees, give or take two, and I am much less sure about next month. Which would you trust with your picnic? Today we teach models to say how sure they are. This is lesson sixty one: probabilistic machine learning.

• Point estimate vs belief
• Uncertainty as output

In the last lesson we finished reinforcement learning. Now we return to an idea from lesson ten, maximum likelihood. There we asked which parameter value makes the observed data most probable, and we kept exactly one answer. That answer is a single number with no sense of doubt attached. Probabilistic machine learning fixes that.

• Lesson 10: maximum likelihood
• One best parameter value

Here is the plan. First, MAP estimation, and a derivation showing that regularization secretly is a prior belief. Second, full Bayesian inference with a coin we compute by hand. Third, Bayesian linear regression with error bars. Fourth, how to approximate hard posteriors. Then Gaussian processes, and a short look at graphical models.

• MAP and priors
• Bayesian inference
• Gaussian processes
• Graphical models

Bayes' theorem from lesson eight works on parameters too. The probability of theta given the data is proportional to the probability of the data given theta, times the probability of theta before seeing data. That last piece is the prior. It encodes what we believed before the evidence arrived, such as weights are probably small.

• Posterior ∝ likelihood × prior
• Bayes' theorem on parameters

MAP estimation keeps a single answer, but picks the peak of the posterior instead of the peak of the likelihood. Take logarithms, and the product becomes a sum. We maximize the log likelihood plus the log prior. The log prior is just an extra term that rewards parameter values we considered plausible beforehand.

• argmax of log likelihood + log prior
• Prior acts as a penalty

Let us derive something important. Suppose targets equal a linear function of the inputs plus Gaussian noise with variance sigma squared. From lesson ten, the negative log likelihood is one over two sigma squared, times the sum of squared residuals, plus a constant. Maximizing likelihood is just ordinary least squares, exactly as in lesson fourteen.

• y = w·x + Gaussian noise
• Noise variance sigma squared

Now put a Gaussian prior on each weight: mean zero, variance tau squared. Its negative log is w squared over two tau squared, plus a constant. Add this to the loss and multiply everything by two sigma squared. We get the sum of squared errors plus sigma squared over tau squared, times the squared norm of w. That is ridge regression, with lambda equal to sigma squared over tau squared.

• w ~ Normal(0, tau squared)
• Penalty: w squared over 2 tau squared
• lambda = sigma² / tau²

Swap the prior for a Laplace distribution, with density proportional to e to the minus absolute w over b. Its negative log is absolute w over b. The same steps give squared error plus two sigma squared over b, times the sum of absolute weights. That is the lasso. The sharp peak at zero explains why lasso pushes many weights to exactly zero.

• Laplace density: peaked at zero
• Penalty: absolute value of w
• lambda = 2 sigma² / b

A quick check with numbers. If the noise variance is four and the prior variance is two, lambda equals four divided by two, which is two. A tighter prior, say variance one half, gives lambda eight and stronger shrinkage. A common mistake is thinking lambda is arbitrary. It is the ratio of noise to prior belief, and tuning it is choosing how much you trust the prior.

• sigma² = 4, tau² = 2 → lambda = 2
• Stronger belief → larger lambda
• Prior is not a free lunch

MAP is useful, but it still throws away information. Two posteriors can share the same peak while one is sharp and the other is wide and flat. The point estimate hides that difference. To keep the whole shape of our belief, we need full Bayesian inference, and the cleanest place to see it is a coin.

• Still a single point
• Ignores posterior width
• Not invariant to reparameterization

We flip a coin ten times and see seven heads and three tails. Let theta be the unknown probability of heads. Maximum likelihood says theta equals seven tenths. But would you really bet that a coin is biased after only ten flips? Let us build a prior that reflects honest uncertainty.

• theta = probability of heads
• 10 flips: 7 heads, 3 tails
• MLE says 0.7

The Beta distribution lives between zero and one, which suits a probability. Its two parameters, alpha and beta, behave like imaginary earlier flips: alpha imaginary heads and beta imaginary tails. Beta of one and one is flat. Beta of two and two is a soft hill centred at one half, which says: probably fair, but I am open to being wrong.

• Beta(alpha, beta) lives on 0 to 1
• Beta(2,2): gentle belief in fairness
• Alpha, beta act like pseudo-counts

Here is the beautiful part. The likelihood of seven heads and three tails is proportional to theta to the seventh, times one minus theta to the third. Multiply by the prior and the exponents simply add. The posterior is Beta with alpha two plus seven, which is nine, and beta two plus three, which is five. Bayesian updating became counting.

• Prior Beta(2,2)
• Data: 7 heads, 3 tails
• Posterior Beta(9,5)

When the posterior belongs to the same family as the prior, we call the prior conjugate to the likelihood. Beta is conjugate to the Bernoulli and binomial. This is why the update was so easy, and why conjugate pairs were the workhorse of Bayesian statistics for decades. Most modern models are not conjugate, and that is where trouble starts.

• Prior and posterior: same family
• Update stays closed-form
• Beta × Binomial = Beta

Now read off the answers. The posterior mean is alpha over alpha plus beta, nine fourteenths, about zero point six four three. The MAP, the mode, is alpha minus one over alpha plus beta minus two, eight twelfths, about zero point six six seven. The variance formula gives roughly zero point zero one five three, so a standard deviation near zero point one two four. The maximum likelihood answer, zero point seven, was pulled toward our prior.

• Mean: 9/14 ≈ 0.643
• MAP: 8/12 ≈ 0.667
• Std dev ≈ 0.124

What if we flip a hundred times and see seventy heads? The posterior is Beta seventy two, thirty two, with mean about zero point six nine, nearly the maximum likelihood answer. The prior had only four imaginary flips, so real data overwhelms it. Bayesian inference is a smooth dial between prior belief and evidence, and the posterior narrows as data grows.

• Prior counts: 4 pseudo-flips
• Real data: 100 flips dominate
• Posterior narrows with data

Now apply the same thinking to regression. Instead of one best line, we keep a distribution over lines. Before data, we draw lines from the prior, and they are all over the place. After seeing points, only lines that pass near the data stay plausible. The spread of those surviving lines is our uncertainty.

• Prior over weights
• Posterior over weights
• Many plausible lines

With a Gaussian prior and Gaussian noise, the posterior over weights is again Gaussian, so this is another conjugate case. Let alpha be the prior precision and beta the noise precision, where precision means one over variance. The posterior covariance S is the inverse of alpha times the identity plus beta times Phi transpose Phi. The posterior mean m equals beta times S times Phi transpose y.

• Prior: w ~ N(0, I / alpha)
• Noise precision: beta
• S = (alpha I + beta ΦᵀΦ)⁻¹
• m = beta S Φᵀ y

Let us compute one by hand. Model: y equals w times x. Set alpha and beta both to one. Data points: x one with y two, and x two with y three. Phi transpose Phi is one plus four, which is five. Phi transpose y is two plus six, which is eight. So S equals one over one plus five, one sixth, and m equals eight sixths, about one point three three three.

• Model: y = w x
• Data: (1, 2) and (2, 3)
• S = 1/6, m = 4/3

To predict at a new input, we average over all plausible weights. The predictive mean is m times x. The predictive variance is one over beta, the irreducible noise, plus x transpose S x, our uncertainty about the weights. At x equals three, the mean is four. The variance is one plus nine over six, which is two point five. Notice the second term grows with distance from the data.

• Variance = 1/beta + xᵀ S x
• Noise + weight uncertainty
• At x = 3: mean 4, variance 2.5

This formula separates two kinds of uncertainty. The noise term is aleatoric: randomness inherent in the world. Collecting more data will not remove it. The weight term is epistemic: uncertainty from not knowing enough. More data shrinks it. Telling these apart is one of the main reasons practitioners reach for Bayesian methods.

• Aleatoric: noise in the data
• Epistemic: lack of knowledge
• More data shrinks only epistemic

Conjugacy is a luxury. In general, Bayes' theorem has a denominator, the evidence, which is an integral over every possible parameter setting. In a neural network with millions of weights, that integral is hopeless to compute exactly. So we approximate, and there are two big families of ideas: sampling and optimization.

• Normalizing constant is an integral
• Non-conjugate models
• Neural networks

Markov chain Monte Carlo builds a random walk whose visited points follow the posterior. In the Metropolis method, we propose a small move. If the new point has higher posterior density, we accept. If lower, we accept with probability equal to the density ratio. A ratio of one half means a coin flip. The normalizing constant cancels in that ratio, which is the whole trick.

• Propose a nearby move
• Accept with min(1, ratio)
• Samples approximate the posterior

Variational inference turns the problem into optimization. Choose a simple family, say Gaussians, and tune its parameters until it is as close as possible to the true posterior, measured by KL divergence from lesson ten. We cannot compute that directly, but maximizing a quantity called the evidence lower bound achieves the same thing. It is fast, but the simple family can miss shape.

• Pick a simple family q
• Make q close to the posterior
• Maximize the ELBO

In practice the trade-off is clear. MCMC is asymptotically exact but can be slow, especially in high dimensions. Variational inference is fast and scales to big models, but it commits to a simple shape and often underestimates the spread of the posterior. Neither is universally better; choose by dataset size and how much you trust the answer.

• MCMC: accurate, slow
• Variational: fast, biased
• Choose by scale and need

Bayesian regression put a distribution on weights. A Gaussian process puts a distribution directly on functions. The definition: for any finite set of input points, the function values have a joint Gaussian distribution. We specify an average function, usually zero, and a kernel that says how function values at two inputs relate.

• Not weights: whole functions
• Any finite set of points is Gaussian
• Defined by a mean and a kernel

From lesson thirty eight, covariance measures how two quantities move together. The kernel is exactly that, for function values. The popular RBF kernel is e to the minus the squared distance over two times length scale squared. Inputs at distance zero have covariance one. At distance one with length scale one, it is e to the minus one half, about zero point six one. Close inputs move together, so functions are smooth.

• k(x, x') = covariance of f(x), f(x')
• RBF: exp(-(x-x')² / 2l²)
• Nearby inputs → similar outputs

Prediction is Gaussian conditioning. Let K be the kernel matrix among training inputs, K star the covariances between training and test points, and K star star among the test points. Given training values y, the predictive mean is K star transpose, times K inverse, times y. The predictive covariance is K star star minus K star transpose, K inverse, K star. The second term subtracts what the data already told us.

• Mean: K*ᵀ K⁻¹ y
• Cov: K** − K*ᵀ K⁻¹ K*
• Just Gaussian conditioning

Time for a real example. One noiseless training point: x equals zero, y equals one. Use the RBF kernel with length scale one. K is just one. Test point x equals one gives K star of e to the minus one half, zero point six zero six five. The mean is zero point six zero six five times one, about zero point six one. The variance is one minus zero point six zero six five squared, which is one minus zero point three six eight, so zero point six three two.

• Train: x=0, y=1
• Test: x=1
• Mean 0.607, variance 0.632

Try x equals two. K star is e to the minus two, about zero point one three five. The mean drops to zero point one three five, and the variance is one minus zero point zero one eight, zero point nine eight two. Far from the data, the process forgets and falls back to the prior. At the training point itself the variance is zero. Uncertainty is small where we have evidence and large where we do not.

• x=2: mean 0.135, variance 0.982
• Far away → back to the prior
• At x=0: variance 0

With many points, the plot shows a mean curve with a band of roughly two standard deviations, pinched at the data and ballooning in gaps. The price is the matrix inverse. Solving with K takes order n cubed operations and n squared memory. At a thousand points that is about a billion operations. At ten thousand, a trillion. So exact Gaussian processes suit small data, and approximations help beyond that.

• Band: mean ± 2 standard deviations
• Inverting K costs O(n³)
• Memory is O(n²)

The kernel is where you encode assumptions: smoothness, periodicity, trends. Its hyperparameters, like the length scale, can be fitted by maximizing the marginal likelihood, which naturally balances fit against complexity. And as we saw in lesson forty five, Gaussian processes are the engine inside Bayesian optimization, choosing where to try the next experiment.

• Kernel choice encodes assumptions
• Hyperparameters fit by marginal likelihood
• Powers Bayesian optimization

A graphical model draws a probability distribution as a diagram. Each node is a random variable, and each arrow means direct influence. The joint distribution factorizes into a product of simple pieces: each variable conditioned on its parents. This turns an enormous joint table into a handful of small ones, and it makes the assumptions visible.

• Nodes: variables
• Arrows: direct dependence
• Joint = product of local factors

You already know one graphical model: Naive Bayes from lesson eighteen. It has a class node with arrows pointing to every feature, and no arrows between features. That picture says the features are independent once the class is known, giving p of y times the product of p of each x given y. Richer graphs relax that assumption. Bayesian networks and hidden Markov models are the next steps.

• Class node → feature nodes
• p(y) × product of p(xᵢ | y)
• Naive = features independent given class

Before we recap, three mistakes to avoid. Do not treat the regularization strength as a magic knob; it equals a ratio of noise to prior variance. Do not confuse the posterior mean with the MAP, as our coin gave zero point six four three versus zero point six six seven. And remember that a predictive band includes observation noise, so it is wider than uncertainty about the function alone.

• Lambda is not arbitrary
• Posterior mean is not MAP
• Bands are not noise-free

Let us recap. MAP adds a log prior to the log likelihood. A Gaussian prior gives L2 regularization and a Laplace prior gives L1. Full Bayesian inference keeps the whole posterior, as the Beta nine five coin showed. Bayesian regression and Gaussian processes turn that posterior into predictions with honest error bars, at a cubic cost for exact Gaussian processes.

• MAP = likelihood + prior
• L2 = Gaussian, L1 = Laplace
• Posterior over weights or functions
• GP cost O(n³)

Next time, lesson sixty two, we ask whether we can trust our models: uncertainty in practice, distribution shift, fairness, interpretability and robustness.

• Lesson 62: Trustworthy ML
• Uncertainty, shift, fairness
• Subscribe for the full course