Eigenvectors, SVD and Positive-Definite Matrices Explained Visually

Push a thousand arrows through a matrix and almost all of them get knocked off course. But a few special arrows only stretch or shrink, staying on their own line. Those arrows hold the secret of the matrix.

• Most vectors change direction
• A few special ones do not

Last episode, we saw matrices as transformations that move the whole plane. Today we dig into their hidden structure: eigenvectors, positive-definite matrices, and the singular value decomposition, which can even compress an image.

• Last time: matrices as transformations
• Today: their hidden structure

An eigenvector of a matrix A is a nonzero vector v where A times v equals lambda times v. The matrix only scales it. The number lambda is the eigenvalue, the stretch factor. Negative lambda flips the arrow, and lambda of one leaves it unchanged.

• A v equals lambda v
• v keeps its line
• lambda is the stretch factor

Watch the whole grid deform. Along the direction one, one, space stretches by three. Along one, minus one, it stretches by one, so it stays put. Every other vector is a mixture of these two, so knowing them tells you what the matrix does everywhere.

• Eigenvectors are the grid's axes
• Everything else is a mix

This is powerful because applying the matrix again and again just multiplies each eigen-direction by its eigenvalue, again and again. Directions with a large eigenvalue dominate, and small ones fade. That idea drives long-run behavior in many algorithms.

• Repeated transformations become easy
• Powers of A act on lambda

How do we find them? Start with A v equals lambda v. Write lambda v as lambda times I times v, and move everything to one side: A minus lambda I, times v, equals zero. We want a nonzero v, so this matrix must squash space, which means its determinant is zero.

• A v = lambda v
• (A − lambda I) v = 0
• Need det = 0

Take A with rows two, one and one, two. Subtract lambda from the diagonal. The determinant is two minus lambda, squared, minus one. Expanding gives lambda squared minus four lambda plus three, which factors into lambda minus one, times lambda minus three. So lambda is one or three.

• A = [[2,1],[1,2]]
• (2−λ)² − 1 = 0
• λ² − 4λ + 3 = 0

Now plug each eigenvalue back in. For lambda equals three, A minus three I has rows minus one, one and one, minus one. That says x equals y, so the eigenvector is one, one. For lambda equals one, we get x equals minus y, so the eigenvector is one, minus one.

• λ = 3: x = y, so (1,1)
• λ = 1: x = −y, so (1,−1)

Verify it in NumPy. Call numpy dot linalg dot eig on A. It returns eigenvalues three and one, and unit-length eigenvectors, pointing along one, one and one, minus one, divided by root two. The order and signs may differ, but the directions match our hand result.

• np.linalg.eig(A)
• Eigenvalues 3 and 1
• Vectors are normalized

Our matrix is symmetric, equal to its own transpose. Symmetric matrices are the friendly ones: their eigenvalues are always real, and their eigenvectors are perpendicular. In machine learning, covariance matrices and Hessians are symmetric, so this case matters a lot.

• Symmetric matrices have real eigenvalues
• Eigenvectors are perpendicular

A symmetric matrix is positive-definite if x transpose A x is greater than zero for every nonzero x. Equivalently, all its eigenvalues are positive. Our example has eigenvalues three and one, so it qualifies. Eigenvalues of zero or more, with some allowed to be zero, give positive semi-definite.

• x^T A x > 0 for every x ≠ 0
• Equivalent: all eigenvalues > 0

Plot x transpose A x as a surface, and a positive-definite matrix gives a bowl with a single lowest point at the origin. The eigenvectors are the bowl's axes, and eigenvalues control how steep each axis is. If one eigenvalue is negative, the surface becomes a saddle instead.

• Positive-definite: bowl
• Mixed signs: saddle
• One minimum

This is why optimization cares. If the Hessian, the matrix of second derivatives, is positive-definite at a point, that point is a local minimum. And when eigenvalues differ wildly, the bowl becomes a long narrow valley, which makes gradient descent zigzag. We will use this soon.

• Hessian positive-definite means local minimum
• Eigenvalue spread slows descent

Eigenvectors have limits. They need a square matrix, and even then they can fail to behave nicely. The singular value decomposition works for any matrix of any shape, which is why it is one of the most useful tools in all of applied mathematics.

• Eigen needs square matrices
• SVD works for any matrix

Here is the picture. Any matrix A can be written as U, times sigma, times V transpose. First V transpose rotates the plane. Then sigma stretches along the axes by the singular values. Finally U rotates again. Every linear transformation is just rotate, stretch, rotate.

• A = U Σ Vᵀ
• Vᵀ rotates
• Σ stretches axes
• U rotates again

The singular values are sorted from largest to smallest, and they measure how much each direction matters. They connect to eigenvalues: the squares of the singular values are the eigenvalues of A transpose A, which is always symmetric and positive semi-definite. That links today's two big ideas.

• Sigma holds σ1 ≥ σ2 ≥ ... ≥ 0
• σ² = eigenvalues of AᵀA

SVD also writes A as a sum of simple rank-one layers: sigma one times u one times v one transpose, plus sigma two times u two times v two transpose, and so on. Keep only the first k layers and you get the best possible rank-k approximation, a result known as the Eckart-Young theorem.

• A = σ1 u1 v1ᵀ + σ2 u2 v2ᵀ + ...
• Keep the top k terms

Try it on a grayscale photo, which is just a matrix of pixel brightness. With five layers you see a blurry outline. With twenty, the shapes are clear. With fifty, it is close to the original. Storing k layers costs about k times m plus n plus one numbers, far fewer than m times n.

• Grayscale image is a matrix
• k = 5, 20, 50
• Storage: k(m + n + 1)

In NumPy, call linalg dot svd to get U, the singular values S, and V transpose. To rebuild with k layers, multiply the first k columns of U, a diagonal of the first k singular values, and the first k rows of V transpose. Compare with the original and the error shrinks as k grows.

• U, S, Vt = np.linalg.svd(A)
• A_k = U[:,:k] @ np.diag(S[:k]) @ Vt[:k]

Here is the payoff. A data cloud's covariance matrix is symmetric and positive semi-definite. Its eigenvectors point along the directions of greatest spread, and its eigenvalues tell how much. That is principal component analysis, and we will derive it fully in the dimensionality reduction episode.

• Covariance matrix is symmetric, PSD
• Top eigenvectors = main directions of data

Recap. Eigenvectors are directions a matrix only scales, found by solving the determinant of A minus lambda I equals zero. Positive-definite matrices have positive eigenvalues and bowl-shaped surfaces. SVD is rotate, stretch, rotate, and keeping its top layers compresses data.

• Eigenvectors: preserved directions
• det(A − λI) = 0
• Positive-definite: bowl
• SVD: rotate, stretch, rotate

Next episode, we start calculus with derivatives from first principles, the tool that lets models learn.

• Next: derivatives from first principles
• Subscribe for the full course