Customer Segmentation and a Recommender From Scratch (Capstone 7-8)

Every online shop faces two questions. Who are my customers, really? And what should I show each of them next? Today we answer both with machine learning, from scratch. First we group customers into meaningful segments, then we build a recommender that predicts what someone will love.

• Customer segmentation
• Recommendation system

In the previous lesson we built spam and text classifiers. Now we move to problems with no obvious label. We will use only public data: the UCI Online Retail transactions for segmentation, and MovieLens ratings for recommendations. We lean on lessons twenty-two and forty-four for clustering theory, so we will not repeat it here.

• Segment with k-means and GMM
• Recommend with neighbors and factors
• Evaluate honestly

Raw retail data has one row per purchased item, with an invoice, a date, a quantity, a price and a customer ID. Clustering needs one row per customer. So the first job is aggregation: group by customer and summarize behavior into a few meaningful numbers.

• Rows are purchases
• We need one row per customer

RFM stands for recency, frequency and monetary value. Recency is how many days since the customer last bought. Frequency is how many separate orders they placed. Monetary is the total money they spent. Take a customer who bought ten days ago, twelve times, spending fifteen hundred in total. That is the vector ten, twelve, fifteen hundred.

• Recency: days since last order
• Frequency: number of orders
• Monetary: total spend

Spending is heavily skewed. Most customers spend a little, and a handful spend enormous amounts. K-means uses distance, so those few giants would pull everything toward them. Taking the logarithm compresses the tail: a spend of one hundred becomes about four point six, and a spend of ten thousand becomes about nine point two. A hundred times more money is now only twice as far.

• Spend is heavy-tailed
• A few whales dominate
• Log transform tames it

Next comes scaling. Recency is measured in days, frequency in counts, money in logged currency. Without scaling, whichever has the biggest numbers wins the distance calculation. The z-score fixes this: subtract the mean, divide by the standard deviation. If mean recency is one hundred days with a standard deviation of fifty, a customer at ten days gets z equal to minus one point eight.

• z = (x - mean) / std
• Mean 0, spread 1

A few traps to avoid. Returns appear as negative quantities and must be handled before you total spend. Rows without a customer ID cannot be assigned to anyone, so drop them. And remember that scaling is part of the model: save the mean and standard deviation, because new customers must be transformed with the same numbers.

• Returns and missing IDs
• Fit scaler on all data only here
• Do not skip the log

As we saw in lesson twenty-two, k-means alternates two steps. Assign each customer to the nearest centroid, then move each centroid to the mean of its members. Repeat until nothing changes. It minimizes the within-cluster sum of squared distances, and it prefers round, similarly sized groups.

• Assign to nearest centroid
• Move centroids to the mean
• Repeat until stable

But how many segments? The elbow method runs k-means for K from one to ten and plots the inertia, the total squared distance to centroids. Inertia always falls as K grows, since more centroids sit closer to points. We look for the bend where gains slow down. Often the elbow is vague, so treat it as a hint rather than an answer.

• Plot inertia against K
• Look for the bend
• Judgment, not proof

The silhouette score judges each point. Let a be its average distance to its own cluster, and b its average distance to the nearest other cluster. The score is b minus a, divided by the larger of the two. If a is two and b is five, the score is three over five, which is zero point six. Near one means well placed, near zero means on a border, negative means probably misassigned.

• a: mean distance inside cluster
• b: mean distance to nearest other
• s = (b - a) / max(a, b)

A Gaussian mixture, from lesson forty-four, gives each segment its own center and elliptical shape, and returns soft probabilities instead of hard labels. A customer might be sixty percent segment one and forty percent segment two. To choose K here we use BIC: the number of parameters times the log of the sample size, minus twice the log-likelihood. Lower is better, because it rewards fit but charges for complexity.

• Gaussians with own shape
• Soft membership probabilities
• BIC = k ln n - 2 ln L

Plot BIC against the number of components. Early on, adding a component improves the likelihood a lot and BIC drops. Later the penalty outweighs the gain, and BIC rises. Suppose BIC bottoms out at four components, while silhouette peaks at three and the elbow hints at four. That disagreement is normal, and the final choice should also consider whether the segments are actionable.

• Fit improves, penalty grows
• Pick the minimum
• Compare with silhouette

We have only three features here, but real segmentation often has dozens. PCA from lesson twenty-three projects the data onto its two strongest directions so we can plot it. Colour each point by its segment. If the colours form separate islands, great. If they form one continuous cloud sliced into pieces, the segments are a convenient partition, not natural groups.

• Project 3D to 2D
• Colour by cluster
• A picture, not proof

Now we interpret. Compute the average recency, frequency and spend for each segment, converting back to real units so a manager can read them. Imagine one segment with recency fifteen days, twenty orders and high spend: the champions. Another with recency three hundred days and one order: lapsed one-time buyers. Names like these turn math into marketing decisions.

• Average R, F, M per cluster
• Convert back to real units
• Give each a plain name

Would the same segments appear tomorrow? Test it. Run k-means with different random seeds and see whether the groups agree. Resample eighty percent of the customers, cluster again, and compare the labelings with the adjusted Rand index, where one means identical and zero means random agreement. If your segments dissolve under a small shake, they are noise.

• Re-run with different seeds
• Resample the customers
• Compare labels with adjusted Rand index

A warning. Clustering always returns clusters, even on random data. The groups depend on your features, your scaling, and your K. It is tempting to invent a story for each cluster, but a story is not evidence. The real test is whether treating segments differently, for example with an email campaign experiment, improves an outcome you care about.

• Clusters are a modeling choice
• Different features, different groups
• Validate by action, not by story

Now the second product. A recommender starts with the user-item matrix: one row per user, one column per item, and each entry is a rating if it exists. MovieLens one hundred thousand has nine hundred forty-three users and one thousand six hundred eighty-two movies, but only one hundred thousand ratings. That means the matrix is more than ninety-three percent empty. Our task is to fill in the blanks.

• Rows: users
• Columns: items
• Mostly empty

Before anything clever, build a baseline: recommend the most popular items, perhaps by average rating with a minimum number of votes. It is surprisingly hard to beat, and it requires no personalization. A movie with a single five-star rating should not outrank one with five hundred ratings averaging four point two, so require a minimum count. Any fancy model must prove it beats this.

• Recommend what everyone likes
• Strong and cheap
• Always beat the baseline

Set up evaluation now, because it shapes everything. If you randomly hide ratings, the model may train on a user's later behavior and be tested on earlier behavior, which never happens in production. Instead, sort by timestamp: train on ratings before a cutoff date and test on ratings after. This mirrors real deployment, and it usually gives a lower, more honest score.

• Random splits leak the future
• Train on the past
• Test on what comes after

Two kinds of metric. RMSE measures rating prediction. If errors are one, minus two, zero and one, the squares are one, four, zero, one, which average to one point five, and the root is about one point two two. For ranking, precision at k asks what fraction of the top k recommendations were relevant, while recall at k asks what fraction of all relevant items we found. With five picks, two hits and eight relevant items, precision is zero point four and recall is zero point two five.

• RMSE for rating prediction
• Precision@k and recall@k for ranking

Collaborative filtering says: people who agreed in the past will agree again. Item-based filtering turns each item into a vector, its column of ratings across users. Two movies liked by the same people will have vectors pointing in similar directions. That geometric picture lets us measure similarity with angles.

• Each movie is a column of ratings
• Similar movies get similar columns

Cosine similarity is the dot product divided by the product of lengths. Take movie A rated five, four, zero, one by four users, and movie B rated four, five, zero, one. The dot product is twenty plus twenty plus zero plus one, which is forty-one. Each vector has length square root of forty-two, so the cosine is forty-one over forty-two, about zero point nine seven six. Nearly the same taste.

• cos = (A . B) / (|A| |B|)
• A = 5,4,0,1 B = 4,5,0,1
• Result: 0.976

Compare with movie C, rated one, zero, five, four. Its dot product with A is five plus zero plus zero plus four, which is nine. Its length is also root forty-two, so the cosine is nine over forty-two, about zero point two one four. Movie C attracts a different audience. Notice that we treated missing ratings as zero, a simplification that works reasonably for sparse data but deserves care.

• C = 1,0,5,4
• Dot with A = 9
• Cosine = 0.214

To predict a user's rating for an item, find the k items most similar to it that this user already rated, then take a similarity-weighted average of those ratings. If the user gave four to a neighbor with similarity point nine and two to one with point three, the prediction is four times point nine plus two times point three, over one point two, which is three point five. Rank unseen items by this score and show the top few.

• Find the k most similar items
• Weighted average of the user's ratings
• Recommend the highest predictions

Neighbors work with raw ratings, which is sparse and noisy. Matrix factorization instead assumes a few hidden tastes explain everything. Each user gets a short vector of how much they like each hidden factor, and each movie gets a vector of how much it contains each factor. The predicted rating is the dot product of the two, exactly the alignment idea from lesson three.

• Hidden tastes: action, romance...
• User vector and item vector
• Rating is their dot product

Formally, the big matrix R is approximated by P times Q transpose, where P has one row per user and Q has one row per item, each with f columns. This resembles the SVD from lesson five, but SVD cannot handle missing entries, so we minimize error only on the known ratings. The loss is the sum of squared errors plus lambda times the squared lengths of the vectors, which prevents overfitting.

• R is approximately P times Q transpose
• P: users by f, Q: items by f
• Minimize squared error on known ratings

Train it by stochastic gradient descent. Take one rating of four. The user vector is one and zero point five, the item vector is two and one. The prediction is two plus zero point five, which is two point five, so the error is one point five. With a learning rate of zero point one, each vector moves along the other: the new user vector is one plus zero point one times one point five times two, which is one point three, and zero point six five.

• p = 1.0, 0.5 q = 2.0, 1.0
• Prediction 2.5, true rating 4
• Error e = 1.5

The item vector moves using the old user vector: two plus zero point one times one point five times one gives two point one five, and one plus zero point one times one point five times zero point five gives one point zero seven five. Now the prediction is one point three times two point one five plus zero point six five times one point zero seven five, about three point four nine. The error fell from one point five to about zero point five one in a single step. Real training repeats this over all ratings for many epochs, with a small lambda shrinking vectors.

• q becomes 2.15, 1.075
• New prediction 3.49
• Error shrank from 1.5 to 0.51

There is an alternative called alternating least squares. If you fix all item vectors, finding the best user vector is just ridge regression from lesson seventeen, with a closed form: Q transpose Q plus lambda times identity, inverted, times Q transpose times that user's ratings. Then fix the users and solve for the items. Each half-step is exact and easy to parallelize, which is why ALS is popular for large systems.

• Fix Q, solve for P exactly
• Fix P, solve for Q
• Each step is least squares

Two practical improvements. Add biases: the prediction becomes the global average, plus a user offset for generous or harsh raters, plus an item offset for universally loved or disliked movies, plus the dot product. Then tune the number of factors and lambda on a validation period, never on the test set. Too many factors with weak regularization will memorize the training ratings.

• Add global mean, user bias, item bias
• Tune f and lambda on validation
• More factors can overfit

Now compare everything fairly. Use the same time-based split, and score the popularity baseline, the item neighbors, and matrix factorization. Report RMSE for ratings and precision and recall at ten for ranking. Be aware that a model can win on RMSE yet lose on ranking, so choose the metric that matches the product: a top ten list is a ranking problem.

• Popularity vs neighbors vs factors
• Same time-based split
• Report RMSE and precision@k

Several pitfalls hide here. Never recommend items the user has already rated. Remember that an unrated movie is not a disliked movie, it is simply unknown, so precision at k understates the true quality. And models tend to favor already popular items, which narrows what users see. Measuring coverage and diversity alongside accuracy helps expose this.

• Exclude items already seen
• Unrated does not mean disliked
• Beware popularity bias

Finally, the cold-start problem. A brand new user has an empty row, and a brand new item has an empty column, so there is nothing from which to learn a vector. Common remedies: show popular items to new users, ask a few onboarding questions, and use content features such as genre or description for new items. Hybrid systems blend content and collaborative signals as data accumulates.

• New user: no history
• New item: no ratings
• Factors cannot be learned from nothing

The two halves of this lesson connect. A segment can serve as a fallback recommender: show new customers what their segment buys most. Both projects also teach the same discipline: engineer features carefully, validate with a split that matches the real world, compare against a simple baseline, and be humble about what the outputs mean.

• Segments can seed cold-start users
• Use shared tools: scaling, validation, honesty

Let us recap. For segmentation, we aggregated transactions into recency, frequency and monetary features, logged the skew, and scaled. We chose K using the elbow, the silhouette and BIC, visualized with PCA, profiled each segment in real units, and tested stability before trusting any of it.

• RFM, log, scale
• K by elbow, silhouette, BIC
• Profile, then stability test

For recommendation, we built the sparse user-item matrix, set a popularity baseline, and split by time. We computed cosine similarity between items, then derived matrix factorization with latent factors trained by SGD and ALS, and judged it with RMSE, precision at k and recall at k. We closed with the cold-start problem and its remedies.

• Matrix, baseline, time split
• Cosine neighbors
• Latent factors by SGD or ALS

In the next capstone, lesson sixty-eight, we classify images with a convolutional network and then build a small transformer language model.

• Next: image classification
• Then a small transformer language model
• Subscribe for the series