PCA, SVD, t-SNE and UMAP: Dimensionality Reduction Explained Visually

A photo of a handwritten digit has seven hundred and eighty-four pixels. A cell in a genetics experiment can have twenty thousand measured genes. Yet you can only look at two or three dimensions. How do you see the shape of data that lives in a thousand dimensions?

• 784 pixels per digit
• 20,000 genes per cell
• Humans see 2D or 3D

Last time we grouped unlabeled points with clustering: k-means, hierarchical, DBSCAN and Gaussian mixtures. Today we attack the sibling problem. Instead of grouping rows, we compress columns. We will build PCA from scratch, connect it to the singular value decomposition, and then meet t-SNE and UMAP.

• Clustering groups rows
• Reduction compresses columns
• Today: PCA, SVD, t-SNE, UMAP

First, a one-scene recap of the curse of dimensionality. As dimensions grow, volume explodes, so a fixed number of points becomes sparse, and distances between points become nearly identical. Nearest neighbours stop being meaningfully near. Models need exponentially more data to cover the space.

• Space grows exponentially
• Data becomes sparse
• Distances concentrate

But real data is rarely spread through all those dimensions. Pixels of a digit are strongly correlated, and genes move together in programs. The data hugs a much lower-dimensional surface inside the big space, like a sheet of paper crumpled inside a room. Dimensionality reduction tries to find that surface.

• Features are correlated
• Intrinsic dimension is small
• Find the surface

There are three common motives. Visualization, where we want a picture. Compression and speed, where fewer features mean cheaper models. And noise removal, where we keep the strong signal directions and drop the weak, noisy ones. Keep these purposes in mind, because they decide which method you should choose.

• Visualization
• Compression and speed
• Noise removal

Here is the core idea of principal component analysis. Imagine a cloud of points shaped like a tilted ellipse. If we must describe each point with a single number, we project it onto a line. Which line? The one along which the points are most spread out, because spread is information.

• Project onto a line
• Spread equals information
• Rotate to maximize variance

There is a second, equivalent view. Each projection has a reconstruction error: the distance from the original point to its shadow. By the Pythagorean theorem, the squared length of a point equals the squared shadow length plus the squared error. The lengths are fixed, so maximizing variance is the same as minimizing reconstruction error.

• Pythagoras links them
• Max variance equals min error
• Two views, one answer

Before anything else, we must center the data. Subtract the mean of each feature so the cloud is centered at the origin. Skipping this is the most common PCA mistake: without centering, the first component simply points toward the mean instead of along the direction of greatest spread.

• Subtract each feature mean
• Uncentered PCA points to the mean
• Most common mistake

Now the formula. Stack the centered data into a matrix X with n rows, the samples, and d columns, the features. The covariance matrix is S equals one over n minus one, times X transpose X. It is d by d. Diagonal entries are feature variances; off-diagonal entries say how pairs of features move together.

• X is n by d, centered
• S = XᵀX / (n−1)
• Diagonal: variances

Pick a unit direction w. The projection of a point x onto it is the dot product w transpose x. The variance of all projections turns out to be w transpose S w. Why? Variance is the average squared projection, and the average of w transpose x x transpose w, with w pulled outside, gives w transpose times the average of x x transpose, times w, which is S.

• Projection is wᵀx
• Variance = wᵀSw
• w comes outside the sum

So the problem is: maximize w transpose S w, subject to w transpose w equals one. We need the constraint, otherwise we could make the variance huge just by scaling w. Constrained optimization calls for a Lagrange multiplier. Form the Lagrangian: L equals w transpose S w, minus lambda times, w transpose w minus one.

• Maximize wᵀSw
• Constraint: wᵀw = 1
• Lagrangian with multiplier λ

Take the gradient with respect to w and set it to zero. The derivative of w transpose S w is two S w, because S is symmetric, and the derivative of lambda w transpose w is two lambda w. So two S w minus two lambda w equals zero, which gives S w equals lambda w. That is exactly the eigenvector equation.

• Gradient: 2Sw − 2λw = 0
• So Sw = λw
• w must be an eigenvector

Which eigenvector? Multiply both sides by w transpose: w transpose S w equals lambda times w transpose w, which equals lambda. So the variance along an eigenvector is exactly its eigenvalue. To maximize variance, choose the eigenvector with the largest eigenvalue. The second component is the next one, automatically perpendicular, since S is symmetric.

• Variance along w equals λ
• Pick the largest λ
• Next components are orthogonal

Let us do it by hand. Four points: two comma one, three comma three, four comma three, and five comma five. The mean of x is three point five, and the mean of y is three. Centered points are minus one point five, minus two; minus zero point five, zero; zero point five, zero; and one point five, two.

• Means: 3.5 and 3
• Subtract from each coordinate
• Centered cloud at origin

Now the covariance, dividing by n minus one, which is three. The variance of x is two point two five plus point two five plus point two five plus two point two five, which is five, over three, about one point six six seven. For y it is eight over three, about two point six six seven. The covariance is six over three, which is two.

• Var x = 5/3 ≈ 1.667
• Var y = 8/3 ≈ 2.667
• Cov = 6/3 = 2

Eigenvalues come from the characteristic equation. The trace is four point three three three, and the determinant is one point six six seven times two point six six seven minus four, which is about zero point four four four. Solving gives lambda one near four point two two eight, and lambda two near zero point one zero five.

• Trace ≈ 4.333
• Determinant ≈ 0.444
• λ₁ ≈ 4.228, λ₂ ≈ 0.105

For the first eigenvector, solve S minus lambda one I, times v, equals zero. The first row gives minus two point five six one v one plus two v two equals zero, so v two is about one point two eight v one. Normalize to length one, and we get roughly zero point six one five, zero point seven eight eight. That is the first principal component.

• v₂ ≈ 1.28 v₁
• Divide by length 1.625
• PC1 ≈ (0.615, 0.788)

Project the first centered point, minus one point five and minus two, onto PC1: minus one point five times point six one five, plus minus two times point seven eight eight, gives about minus two point five. The first component captures four point two two eight out of four point three three three total variance, which is ninety-seven and a half percent. One number per point keeps almost everything.

• Score of point 1 ≈ −2.5
• PC1 variance 4.228 of 4.333
• About 97.6 percent kept

Forming X transpose X squares the numbers, which can lose precision. The singular value decomposition avoids this. Any matrix X factors as U times Sigma times V transpose. U and V have orthonormal columns, and Sigma is a diagonal matrix of non-negative singular values, sorted from largest to smallest.

• X = UΣVᵀ
• Orthonormal U and V
• Singular values sorted

Why does this give PCA? Substitute the factorization into the covariance. X transpose X becomes V Sigma transpose U transpose U Sigma V transpose, and since U transpose U is the identity, we get V Sigma squared V transpose. That is an eigendecomposition. The columns of V are the principal directions, and the eigenvalues equal sigma squared over n minus one.

• XᵀX = V Σ² Vᵀ
• V holds the components
• λᵢ = σᵢ² / (n−1)

The scores, meaning the data in the new coordinates, are X times V, which equals U times Sigma. To keep k components, take the first k columns of V and project. This is also the best rank k approximation of X, by the Eckart-Young theorem, in the least squares sense. That is why SVD is the numerically preferred way to compute PCA.

• Scores = XV = UΣ
• Keep first k columns
• Eckart-Young: best rank-k fit

How many components should we keep? The explained variance ratio of component i is its eigenvalue divided by the sum of all eigenvalues. Plot these in a scree plot, and also plot their cumulative sum. A common rule of thumb is to keep enough components to reach ninety or ninety-five percent, but that is a heuristic, not a law.

• Ratio = λᵢ / Σλ
• Scree plot and elbow
• 90 to 95 percent is a heuristic

Here is a trap. PCA chases variance, and variance depends on units. If one feature is height in millimetres and another is age in years, the millimetre feature dominates the first component purely because its numbers are bigger. Standardize features to zero mean and unit variance first, unless the units are genuinely comparable.

• Variance depends on units
• Large-scale feature dominates
• Standardize first

Whitening goes one step further. After projecting onto the components, divide each score by the square root of its eigenvalue. Now every component has variance one and they are uncorrelated, so the covariance matrix becomes the identity. The ellipse becomes a circle. Whitening helps some algorithms, but it also amplifies noise in tiny-variance directions.

• Divide scores by √λ
• Unit variance, uncorrelated
• Amplifies tiny noisy directions

Reconstruction closes the loop. To go back, multiply the scores by V transpose, then add the mean again. With all components you recover the data exactly; with k components you get a smoothed approximation. The mean squared reconstruction error equals the sum of the discarded eigenvalues, a neat accounting of what you threw away.

• Back: scores · Vᵀ + mean
• Error = discarded eigenvalues
• More components, sharper image

Picture three-dimensional data shaped like a flat pancake tilted in space. PCA finds an orthonormal rotation of the axes. The first two new axes lie in the pancake, and the third points through its thin direction. Dropping that third axis flattens the pancake with almost no loss. PCA is simply a rotation followed by throwing axes away.

• Rotate to principal axes
• Drop low-variance axes
• 3D pancake becomes 2D

Time to code it with only NumPy. Step one, compute the mean of each column and subtract it. Step two, call numpy linalg svd on the centered matrix, with full matrices set to false. Vt holds the components as rows. Step three, the scores are the centered data times the transpose of the first k rows of Vt.

• Subtract column means
• np.linalg.svd on centered data
• components_ = Vt[:k]

The explained variance is S squared divided by n minus one, and the ratio divides that by its total. Transform is centered data times components transpose. Inverse transform is scores times components, plus the mean. Run it on our four points and you should see the first variance near four point two two eight and a ratio near zero point nine seven six.

• variance = S² / (n−1)
• inverse = scores @ components + mean
• Matches our hand calculation

Two practical notes. First, signs are arbitrary: an eigenvector and its negative are equally valid, so your components may be flipped compared to scikit-learn. Second, always fit PCA on the training set only, then apply it to validation and test data. Fitting on everything leaks information, a mistake we met in the earlier episodes on leakage.

• Eigenvector signs are arbitrary
• Fit on training data only
• Use a pipeline

PCA is linear. It can only rotate and drop axes. If the data lies on a curved surface, like a Swiss roll, the direction of greatest variance cuts straight through the roll and squashes different layers on top of each other. Neighbours that were far apart along the roll become neighbours in the projection.

• PCA only rotates and drops
• Swiss roll gets squashed
• Need nonlinear methods

t-SNE, short for t-distributed stochastic neighbour embedding, takes a different view. It does not maximize variance. It tries to preserve neighbourhoods. Points that are close in the original space should stay close in the two-dimensional map, and points that are far can be anywhere reasonably far.

• Not variance, but neighbours
• Close stays close
• Far can be loosely far

Step one, in the original space, turn distances into probabilities. For point i, put a Gaussian on it and say point j is its neighbour with probability proportional to e to the minus squared distance over two sigma squared, normalized across all j. Sigma is chosen per point so that the effective number of neighbours matches a setting called perplexity.

• p(j|i) from Gaussian similarity
• Sigma set per point
• Perplexity: effective neighbours

Step two, do the same in the low-dimensional map, but with a Student t distribution with one degree of freedom, which has heavy tails. Step three, move the map points to make the two probability distributions match, by minimizing the KL divergence with gradient descent. Heavy tails give moderately distant points room, which relieves crowding and separates clusters.

• Student t in the map
• Minimize KL divergence
• Heavy tails reduce crowding

The KL divergence is asymmetric. Placing true neighbours far apart is heavily punished, while placing non-neighbours close together is punished lightly. So t-SNE guards local structure fiercely, but it has little incentive to get global structure right. That single fact explains most of its pitfalls.

• Local structure protected
• Global structure ignored
• Source of most pitfalls

UMAP, uniform manifold approximation and projection, shares the spirit but differs in machinery. It builds a weighted graph of each point's nearest neighbours, where edge weights say how strongly points are connected, using local distance scaling so each point sees roughly the same number of neighbours. Think of it as a fuzzy neighbourhood graph.

• k-nearest-neighbour graph
• Local distance scaling
• Edge weights as fuzzy strength

Then UMAP lays out a low-dimensional graph with the same connectivity. It starts from a spectral initialization, then uses attractive forces along edges and repulsive forces between random non-neighbours, optimizing a cross-entropy between the two graphs. Because it uses sampled repulsion, it is usually faster than t-SNE on large datasets.

• Spectral initialization
• Attract edges, repel randoms
• Faster on large data

Two parameters matter most. N neighbours controls locality: small values emphasize fine detail, large values emphasize broad structure. Min dist controls how tightly points may pack: small values make clumpy clusters, large values spread them out. UMAP often keeps more global arrangement than t-SNE, though that is a tendency, not a guarantee.

• n_neighbors: local versus global
• min_dist: packing tightness
• More global, not guaranteed

Now the pitfalls, because these plots are easy to over-read. First, cluster sizes mean little. t-SNE expands dense clusters and shrinks sparse ones, so a big blob does not imply a big group. Second, distances between clusters mean little. Two clusters far apart might not be more different than two that are close.

• Cluster size is unreliable
• Gaps between clusters unreliable
• Do not measure with a ruler

Third, results depend on hyperparameters and randomness. With low perplexity, a single blob can fragment into fake clusters. With some settings, random noise can show convincing structure. Run several perplexities and several seeds. If a structure appears only under one setting, be suspicious. Real structure survives changes.

• Try multiple perplexities
• Try multiple seeds
• Trust only stable structure

Fourth, axes have no meaning. In PCA, axis one is a direction with a defined weight on each feature. In t-SNE and UMAP, the axes are arbitrary, and rotating or flipping the map is equally valid. Also, you generally cannot embed new points consistently with t-SNE, while UMAP and PCA offer a transform step.

• PCA axes have loadings
• Embedding axes arbitrary
• New points: PCA yes, t-SNE no

A common and costly mistake is to run t-SNE or UMAP, cluster the two-dimensional picture, and treat the result as truth. These embeddings are visualization tools. They discard information, distort geometry, and depend on settings. Cluster in the original or in a PCA-reduced space, and use the map to inspect the result.

• Maps are for looking
• Cluster before embedding
• Inspect results on the map

For prediction, PCA is a legitimate preprocessing step. It can reduce noise, speed up training, and tame correlated features. But it is unsupervised: the top variance directions are not guaranteed to be the most predictive. A low-variance direction may be exactly the one that separates your classes. Always compare with and without PCA on validation data.

• PCA ignores the labels
• Low-variance may separate classes
• Validate with and without

A practical recipe follows. Standardize, then run PCA, keeping enough components to explain most variance, perhaps fifty for something like image data. Feed those into your model. For a picture, run UMAP or t-SNE on the PCA output, which is faster and less noisy. Always do this inside a pipeline fit only on training data.

• Standardize, then PCA
• Model on PCA features
• UMAP on PCA output for plots

Let us recap. PCA finds orthogonal directions of maximum variance. The Lagrange multiplier turns that into an eigenvector problem of the covariance matrix, and the SVD solves it stably. Explained variance guides how many components to keep, standardizing prevents unit bias, and whitening equalizes variances.

• Max variance equals eigenvector
• SVD is the stable route
• Scale, scree, whiten

t-SNE and UMAP preserve neighbourhoods rather than variance, giving excellent pictures of local structure. But cluster size, between-cluster distance and axes are not reliable, and settings matter. Use them to explore, not to prove.

• Maps: explore, do not prove
• Next: model evaluation
• Subscribe for episode 24