Matrices as Transformations: See Linear Algebra Move

A matrix looks like a boring grid of numbers. But every matrix is secretly a machine that takes all of space and bends it. Once you see that, half of machine learning starts to make sense.

• A grid of numbers
• Actually a transformation

Last time we met vectors and dot products. Today we learn what matrices do to those vectors, how to combine them, and how to undo them. We will also check everything in NumPy.

• Shape, transpose, identity
• Transformations and products
• Determinant, inverse, rank

A matrix is a rectangle of numbers. A matrix with two rows and three columns has shape two by three, always rows first. A dataset with one hundred samples and five features is a one hundred by five matrix.

• Rows first, then columns
• Dataset: samples by features

The transpose flips a matrix across its diagonal, turning rows into columns. Our two by three matrix becomes three by two. Adding matrices works entry by entry, and only when the shapes match exactly.

• Transpose swaps rows and columns
• Add only equal shapes

The identity matrix has ones on the diagonal and zeros elsewhere. It is the do nothing matrix: multiply any vector by it and nothing changes. It plays the role that the number one plays in ordinary arithmetic.

• Ones on the diagonal
• Changes nothing

Here is the big idea. The columns of a matrix tell you where the basis vectors land. Take the matrix with columns zero comma one, and negative one comma zero. The first basis vector jumps up, the second swings left, and the entire grid rotates ninety degrees.

• Column one is where i lands
• Column two is where j lands

Change the columns and you change the motion. A diagonal matrix with two and three scales the grid, stretching x by two and y by three. A matrix with a one in the upper right corner shears it, sliding the grid sideways while keeping the x axis fixed.

• Diagonal matrix scales
• Off-diagonal entry shears

So matrix vector multiplication is just this. Write the vector as x times the first basis vector plus y times the second. After the transformation, it becomes x times the first column plus y times the second column. Every point follows the grid.

• Mix the columns
• Weights are the vector's entries

Multiplying two matrices means doing one transformation after another. Rotate, then shear, and the combined effect is a single new matrix. Order matters: shear then rotate gives a different result, so A times B is usually not B times A.

• Apply one, then the other
• Order matters

By hand, each entry of the product is a row of the first matrix dotted with a column of the second. Take one two, three four times five six, seven eight. The top left entry is one times five plus two times seven, which is nineteen.

• Row dot column
• Top left: 1x5 + 2x7 = 19

The shape rule keeps you safe. An m by n matrix times an n by p matrix gives an m by p matrix. The inner numbers must match, and they vanish. If they do not match, the product simply does not exist.

• Inner sizes must match
• Outer sizes give the result

The determinant tells you how much a transformation scales area. The unit square has area one. The scaling matrix with two and three turns it into a rectangle of area six, so its determinant is six. The shear keeps area the same, so its determinant is one.

• det = ad - bc
• Scale gives 6, shear gives 1

A negative determinant means space was flipped, like a mirror. And a determinant of zero is special: it means space got squashed onto a line or a point. Area collapses to nothing, and that information can never be recovered.

• Negative: space flips
• Zero: space collapses

The inverse matrix is the transformation that undoes another one. Apply a matrix, then its inverse, and you land back where you started, which is the identity. Only matrices with a nonzero determinant have an inverse, because you cannot unsquash a flattened plane.

• Undoes the transformation
• Exists only if det is not 0

This gives us a way to solve linear systems. The equations can be written as A x equals b: which input vector x lands on b after the transformation? If A has an inverse, the answer is x equals A inverse times b.

• Which x lands on b?
• x = A inverse b

The span of some vectors is every point you can reach by scaling and adding them. Two vectors pointing in different directions span the entire plane. But if one vector is just a multiple of the other, they are linearly dependent, and their span is only a line.

• Span: all reachable points
• Dependent vectors add nothing new

A basis is a set of independent vectors that span the space, a minimal set of directions to describe everything. Rank counts how many independent columns a matrix has, which is the dimension of its output. Full rank means no squashing, while lower rank means information is lost.

• Basis: minimal independent directions
• Rank: dimension of the output

NumPy checks all of this in a line or two. The at symbol multiplies matrices, dot T gives the transpose, linalg dot det gives the determinant, linalg dot inv the inverse, and linalg dot solve solves systems. Use solve rather than inv when you can, since it is faster and more stable.

• @ for products, .T for transpose
• Prefer solve over inv

Now the machine learning link. Stack your dataset into a matrix X with one row per sample and one column per feature. Put the weights in a vector w. Then X times w gives the predictions for every sample at once, with no loop.

• Rows are samples
• One product, all predictions

That single product is the engine of linear regression, and later of every neural network layer. Each layer is a matrix transformation of your data, followed by a bend. Understanding matrices as moving space is the key to understanding what models do.

• Regression uses X w
• Neural layers transform data

Let us recap. Matrices have shapes, transposes and an identity. Their columns show where the basis vectors land, so they rotate, scale and shear space. Products compose transformations, the determinant scales area, and the inverse undoes a map when the determinant is not zero.

• Columns show where basis lands
• Determinant scales area
• Inverse undoes the map

Span, independence, basis and rank tell us how much of space a matrix can reach. And X times w makes predictions for a whole dataset at once. Next episode, we find the special directions a matrix leaves unchanged: eigenvectors, plus the singular value decomposition.

• Directions that stay put
• Subscribe for episode 5