Vectors and Dot Products: The Geometry Behind Every ML Prediction

Every time a model predicts a house price, a spam score, or the next word, it does one tiny operation millions of times. That operation is the dot product.

• The dot product powers predictions

Last time we covered functions, exponents, logarithms and sigma notation. Today we start linear algebra, and by the end you will see why the dot product measures alignment.

• Episode 3: vectors
• Next: matrices

A scalar is a single number, like a temperature of twenty degrees. A vector is an ordered list of numbers, like the size, bedrooms and age of a house.

• Scalar: one number
• Vector: ordered list

A matrix is a grid of numbers, and a tensor is the general idea of an array with any number of axes. The shape tells you the size along each axis. A vector with three entries has shape three.

• Scalar, vector, matrix, tensor
• Shape = size per axis

A vector can be drawn as an arrow from the origin, or as a point at its tip. In machine learning it is also one row of data: one example described by its features.

• Arrow from the origin
• Or one row of data

To add vectors, add matching entries. Three and one comma two and four gives four comma six. Visually, place the second arrow at the tip of the first, and the sum goes from start to finish.

• Add entries one by one
• Tip to tail

Multiplying a vector by a scalar stretches or shrinks it. Multiply by two and it doubles in length. Multiply by negative one and it flips direction. The direction stays on the same line.

• Scalar stretches or shrinks
• Negative flips direction

The length, or norm, of a vector comes from the Pythagorean theorem. For three comma four, square each entry, add to get twenty five, and take the square root to get five.

• Norm = square root of sum of squares
• (3, 4) has length 5

The distance between two points is the norm of their difference. Subtract the vectors, then take the length. This is how a nearest neighbor method decides which examples are similar.

• Distance = norm of difference
• Used in nearest neighbors

The dot product multiplies matching entries and adds the results. Take one comma two comma three with four comma negative one comma two. That is four, minus two, plus six, which equals eight. One number comes out.

• Multiply matching entries
• Add them: one number

Geometrically, the dot product equals the length of a, times the length of b, times the cosine of the angle between them. Same direction gives a large positive value. Opposite directions give a negative one.

• a·b = ||a|| ||b|| cos θ
• Aligned: big positive
• Opposite: negative

Think of a shadow. Shine light straight down onto vector a, and the shadow of b along a has length b times cosine theta. The dot product is that shadow length times the length of a.

• Shadow of b onto a
• Dot product = shadow times length

If we divide the dot product by both lengths, we get cosine similarity, a number between negative one and one. It ignores size and only compares direction, which is why it is popular for comparing text and embeddings.

• Divide by both lengths
• Range: -1 to 1
• Compares direction only

Two vectors are orthogonal when their dot product is zero, meaning they meet at a right angle. One comma two and two comma negative one give two minus two, which is zero. Orthogonal vectors share no alignment at all.

• Dot product zero
• Right angle
• No shared direction

In NumPy, we build arrays with np dot array, then call np dot of a and b to get the dot product. Use np linalg norm for length. Always check shapes first, because mismatched shapes cause errors.

• np.dot(a, b)
• np.linalg.norm(a)
• Check shapes first

Here is the payoff. A model holds a weight vector w and a bias b. Given a feature vector x, its prediction is w dot x plus b. Each weight says how strongly one feature pushes the answer up or down.

• Prediction = w·x + b
• Weights scale each feature
• Bias shifts the result

Geometrically, the weight vector is a direction in feature space. The prediction is large when the data point aligns with that direction. Learning a model means finding the weight vector that aligns best with the answers.

• w is a direction
• Prediction = alignment with w

Let us recap. Vectors are lists of numbers, drawn as arrows or read as data rows. We add and scale them, measure length with the norm, and find distance by subtracting. The dot product multiplies and sums, measuring alignment and projection. Zero means orthogonal.

• Vectors, norms, distances
• Dot product = alignment
• Zero means orthogonal

Next episode, matrices as transformations: grids of numbers that stretch, rotate and reshape space.

• Next: matrices
• Subscribe for the series