Algebra Essentials for Machine Learning: Functions, Exponents, Logs and Sigma

Every machine learning model, no matter how fancy, is built from a handful of algebra ideas. Miss one, and the formulas look like hieroglyphics. Learn them now, and they read like plain sentences.

• A few ideas, everywhere
• Learn them once

In the last episode we saw that machine learning means learning a rule from data by shrinking a loss. Today we build the language that lets us write that rule down. By the end, you will read a full loss formula symbol by symbol.

• Learning shrinks a loss
• Today: write rules down

A variable is a named box that can hold different numbers. A constant is a box whose value never changes. In an expression like w times x plus b, x is the input, while w and b are numbers the model will adjust.

• Variable: changing box
• Constant: fixed box

A function is a machine: put a number in, get exactly one number out. We write f of x equals w times x plus b. If w is two and b is one, then f of three is seven.

• One input, one output
• f(x) = wx + b

This is the same idea as y equals m x plus b from school. The slope tells you how much the output rises when the input goes up by one. The intercept b is where the line crosses the vertical axis, the output when x is zero.

• m (or w): slope
• b: intercept

Slide w up and the line tilts steeper. Make w negative and it falls. Slide b and the whole line shifts up or down without tilting. Training a model is just a search for the best values of these two sliders.

• w tilts the line
• b shifts the line

In machine learning, w is called a weight and b a bias. Predicting a house price from its size could be price equals w times size plus b. Learning means finding the w and b that fit the data.

• w is the weight
• b is the bias

An exponent counts repeated multiplication. Two to the power of three means two times two times two, which is eight. The big number is the base, and the small raised number is the exponent.

• 2 to the 3 = 2 x 2 x 2 = 8
• Base and exponent

Three laws do most of the work. Multiplying the same base adds the exponents: a to the m times a to the n equals a to the m plus n. Dividing subtracts them. Raising a power to a power multiplies them. Also, anything to the power zero is one, and a negative exponent means a reciprocal: two to the minus three is one eighth.

• Multiply: add exponents
• Divide: subtract
• Power of power: multiply

A root undoes a power. The square root of nine is three, because three squared is nine. Roots are also exponents: the square root of x is x to the power one half. That fact will matter when we measure distances later.

• Square root of 9 is 3
• Root = power of one half

Now flip the roles and make the exponent the variable. The function two to the x doubles every time x grows by one. The special base e, about two point seven one eight, gives the exponential function e to the x, which appears in growth, probabilities, and the sigmoid curve.

• Exponent becomes the variable
• e is about 2.718

A logarithm asks: what exponent do I need? Log base two of eight equals three, because two to the third is eight. The log undoes the exponential, and the exponential undoes the log. The natural log, written ln, uses base e.

• log2(8) = 3
• Asks: which exponent?
• ln uses base e

Here is the superpower. The log of a times b equals the log of a plus the log of b. Multiplication becomes addition. Likewise, the log of a to the n equals n times the log of a, and the log of a over b is a difference.

• log(ab) = log a + log b
• log(a^n) = n log a

Why does machine learning care? Model likelihoods multiply many small probabilities together. Multiply a thousand numbers below one and a computer rounds the result to zero. Take logs and you add moderate negative numbers instead, which is stable. And since the log always increases, the best parameters stay the same.

• Many small probabilities
• Product underflows to zero
• Sum of logs is stable

Sigma, the Greek capital letter, means add up. Under it, i equals one gives the start. On top, the end value. To the right is the recipe. Sigma for i from one to four of i squared is one plus four plus nine plus sixteen, which is thirty.

• Start: i = 1
• Stop: n
• Recipe: i squared

Capital pi does the same with multiplication. Pi for i from one to three of i is one times two times three, which is six. This is how the likelihood of many data points is written. Take the log, and the product becomes a sigma.

• Capital pi multiplies
• Likelihoods are products
• Log turns pi into sigma

Absolute value strips the sign and measures distance from zero. The absolute value of minus four is four. In machine learning, it measures how far off a prediction is, ignoring whether it was too high or too low.

• Distance from zero
• |-4| = 4
• Error ignoring sign

Rearranging a formula means doing the same operation to both sides to keep the balance. From y equals w x plus b, subtract b from both sides, then divide both sides by w, to get x equals y minus b, all over w.

• Same operation both sides
• Solve y = wx + b for x

Now the big one. L equals one over n, times the sum for i from one to n of the quantity y sub i minus y hat sub i, squared. L is the loss, one number scoring how wrong the model is. n is the number of data points. y sub i is the true answer for point i. And y hat sub i, said y-hat, is the model's prediction.

• L: the loss score
• n: number of points
• y: truth, y-hat: prediction

Read it from the inside out. Subtract prediction from truth to get the error. Square it, so negatives cannot cancel positives and big mistakes count more. Sum over all the points. Then divide by n to get an average, so the score does not grow just because you have more data.

• Subtract: the error
• Square: no cancelling, big errors hurt
• Sum, then divide by n

Take three points. True values are three, five, and eight. Predictions are two, five, and ten. The errors are one, zero, and minus two. Squared, they are one, zero, and four. The sum is five. Divide by three, and the loss is about one point six seven.

• Truth: 3, 5, 8
• Prediction: 2, 5, 10
• Loss is about 1.67

Check it in Python. Set y equals np dot array of three, five, eight, and y hat equals np dot array of two, five, ten. Then np dot mean of the quantity y minus y hat, squared, prints one point six six six seven. It matches our hand calculation.

• np.mean((y - y_hat)**2)
• Prints 1.6667

Four common mistakes. Reading the subscript i as a power, when it is only an index. Believing the log of a plus b equals log a plus log b, which it does not. Mixing up minus x squared with the square of minus x; they differ. And dropping the parentheses in the loss before squaring.

• Subscript i is not a power
• log(a+b) is not log a + log b
• -x² differs from (-x)²

To recap: a function turns inputs into outputs, and w x plus b is a line you can tune. Exponents count multiplication, logs undo them and turn products into sums, and sigma and pi write long sums and products compactly. You can now read the loss formula.

• Functions, exponents, logs
• Sigma, pi, absolute value
• Next: vectors and dot products