Regularization and Feature Engineering: Ridge, Lasso and Leak-Free Pipelines

Here is a strange fact. You can add more features to a model, watch the training error drop to almost zero, and still end up with a model that is worse at predicting anything new. Why does that happen, and what can we do about it? This episode is the answer.

• Training error keeps falling
• Test error eventually rises

In the previous episode we learned how to judge a classifier with confusion matrices, precision, recall, thresholds and calibration. Now we turn to the other side of the problem: how to prepare the inputs and how to constrain the model, so that those metrics look good on data the model has never seen.

• Last time: classification metrics
• Today: shaping the model

We will move in three steps. First, feature engineering: polynomial features, categorical encoding, scaling, and the trouble caused by correlated features. Second, regularization: ridge, lasso and elastic net, including a full derivation and the geometry that explains why lasso produces zeros. Third, pipelines, so none of this leaks information.

• Feature engineering
• Regularization and geometry
• Leak-free pipelines

A linear model is only linear in its features. If the true relationship between x and y is curved, a straight line will miss it. But if we hand the model a new column containing x squared, the same linear machinery can draw a curve. We did not change the algorithm. We changed what it gets to look at.

• Linear model, nonlinear data
• Create new columns

Take one row with a equal to two and b equal to three. Degree two polynomial expansion produces six values: one, a, b, a squared, a times b, and b squared. That is one, two, three, four, six, nine. Notice the product a times b, which lets the model capture interactions, where the effect of one feature depends on the other.

• a and b become 1, a, b
• Plus a squared, ab, b squared

The cost is growth. With ten original features, a degree two expansion gives sixty-six columns including the constant. Degree three gives two hundred eighty-six. Degree four gives over a thousand. If you only have a few hundred rows, the model suddenly has more knobs than data, and that is the recipe for overfitting.

• 10 features, degree 2: 66
• 10 features, degree 3: 286

Watch what happens as we raise the degree on fifteen noisy points. Degree one underfits. Degree three follows the trend nicely. Degree fifteen passes through every point and swings wildly in between. Look at the coefficients of that last model: they are enormous, with huge positive and negative values cancelling each other. Remember that clue.

• Degree 1: underfit
• Degree 3: good fit
• Degree 15: wild wiggles

A common mistake is to expand polynomial features and forget the scale. If x runs up to one hundred, then x cubed runs up to one million. The columns live on wildly different scales, which makes optimization hard and, as we will see, makes penalties unfair. Expand first, then standardize, and keep the degree as a hyperparameter you tune.

• Expand, then scale
• Do not expand blindly

Models multiply and add numbers, so a column containing red, green and blue cannot be used directly. We need an encoding. The choice matters, because the encoding quietly tells the model what relationships between categories exist.

• Text categories fail
• Encoding translates them

Suppose we map red to zero, green to one and blue to two. A linear model now believes blue is twice as far from red as green is, and that the effect of blue is exactly double that of green. For colors that is nonsense. This integer trick is only safe when the categories truly have an order, like small, medium and large.

• Red 0, green 1, blue 2
• Implies blue is twice green

One-hot encoding fixes this by creating one column per category. A red row becomes one, zero, zero. Green becomes zero, one, zero. Blue becomes zero, zero, one. Now each color gets its own weight, with no false ordering. The price is more columns, which is painful when a feature has thousands of categories.

• One column per category
• Exactly one 1 per row

Two traps here. First, the three color columns always add up to one, which exactly duplicates the intercept column. In ordinary least squares that makes the problem unsolvable, so we drop one category. Second, target encoding replaces a category with the average label for that category. It is powerful, but if computed on the full data it leaks the answer into the feature, so it must be fitted only on training folds.

• Drop one column when unpenalized
• Target encoding must be inside CV

Imagine a feature measured in years, between twenty and sixty, and another measured in dollars, between thirty thousand and two hundred thousand. Gradient descent sees a long thin valley and zigzags. Even worse, a penalty on the size of coefficients would punish the weights of these features very differently, simply because of their units. Scaling puts every feature on a common footing.

• Age in years, income in dollars
• Penalties depend on scale

Standardization subtracts the mean and divides by the standard deviation. Take heights of one fifty, one sixty, one seventy, one eighty and one ninety. The mean is one seventy. The squared deviations are four hundred, one hundred, zero, one hundred, four hundred, averaging two hundred, so the standard deviation is about fourteen point one. The value one ninety becomes twenty divided by fourteen point one, about one point four one.

• z = (x - mean) / std
• Mean 0, std 1

Min-max scaling maps the smallest value to zero and the largest to one, using x minus min over max minus min. Our height of one seventy becomes point five. It is simple, but fragile. If one outlier at four hundred appears, every normal value is squashed into the bottom sixth of the range. Standardization copes better, and robust scalers based on the median and interquartile range cope better still.

• (x - min) / (max - min)
• One outlier squashes the rest

Here is the rule that will matter again at the end of the video. The mean and standard deviation must be computed from the training data only. Then those same numbers are applied to validation and test data. The test set must never help decide how to transform itself, because that quietly leaks information about data the model is supposed not to have seen.

• Fit on train only
• Reuse train mean and std

Multicollinearity means that features are strongly correlated with each other. Think of house size in square feet and in square meters. They carry the same information. The model can split the credit between them in infinitely many ways, and the loss surface develops a long flat valley where many different weights fit almost equally well.

• Two features say the same thing
• Weights become unstable

Suppose x2 is almost a copy of x1, and the true relationship is y equals x1 plus x2. A fit might return a weight of one on each. Add a little noise and refit, and you might get fifty on x1 and minus forty-eight on x2. The predictions are nearly the same, because the two huge weights cancel. But the individual coefficients are meaningless and swing wildly.

• Truth: y = x1 + x2
• Fit: 50 x1 and minus 48 x2

Recall the normal equation from episode fourteen: w equals the inverse of X transpose X, times X transpose y. With correlated columns, X transpose X has one eigenvalue close to zero. Its inverse has the reciprocal, a huge number, which amplifies any noise in y. That is the algebra behind those unstable weights. In the extreme case of exact duplicates, the inverse does not exist at all.

• X transpose X nearly singular
• Its inverse explodes

To detect it, regress each feature on all the other features and record the R squared. The variance inflation factor is one divided by one minus that R squared. If the other features explain ninety percent of a feature, the factor is one over point one, which is ten. Values above five or ten are a common rule of thumb for trouble. Correlation matrices help too, but VIF also catches combinations of several features.

• VIF = 1 / (1 - R squared)
• Above 5 or 10 is a warning

Regularization adds a price tag for large weights. Ridge regression minimizes the usual sum of squared errors plus lambda times the sum of squared weights. Here y is the target vector, X is the data matrix, w is the weight vector, and lambda is a nonnegative hyperparameter we choose. The intercept is normally left out of the penalty. A large lambda forces small weights, and a lambda of zero gives back ordinary least squares.

• Loss + lambda times sum of w squared
• Lambda controls the tradeoff

Now the derivation. Write the squared norm as a dot product: y minus Xw, transposed, times y minus Xw, plus lambda times w transposed w. Expanding gives y transposed y, minus two times w transposed X transposed y, plus w transposed X transposed X w, plus lambda w transposed w. We expanded carefully so that every term is ready to be differentiated.

• Expand the squared norm
• Differentiate with respect to w

Take the gradient with respect to w. The first term gives zero. The second gives minus two X transposed y. The third gives two X transposed X w. The last gives two lambda w. Set the sum to zero and divide by two: X transposed X w plus lambda w equals X transposed y. Factor out w, writing lambda w as lambda times the identity matrix times w. We get X transposed X plus lambda I, times w, equals X transposed y, so w equals the inverse of that matrix times X transposed y.

• Gradient: -2 X^T y + 2 X^T X w + 2 lambda w
• Set to zero

Why does this fix multicollinearity? Adding lambda times the identity raises every eigenvalue of X transposed X by exactly lambda. X transposed X is positive semi-definite, so its eigenvalues are at least zero, and after the shift they are at least lambda, which is positive. The matrix is therefore always invertible. In our earlier example, the tiny eigenvalue point one becomes one point one, and its inverse drops from ten to under one.

• Eigenvalues shift up by lambda
• Always invertible when lambda > 0

Let us check with numbers. Suppose we have one feature, with X transposed X equal to ten and X transposed y equal to twenty. With lambda zero, the weight is twenty over ten, which is two. With lambda equal to ten, the weight is twenty over ten plus ten, which is twenty over twenty, which is one. Penalty shrank the weight by half. As lambda grows toward infinity, the weight approaches zero but never exactly reaches it.

• X^T X = 10, X^T y = 20
• lambda 0: w = 2; lambda 10: w = 1

If we plot every coefficient as lambda increases, we get the ridge path. Each curve shrinks smoothly toward zero. Correlated features shrink together and share the weight sensibly, instead of one blowing up and the other compensating. Importantly, ridge shrinks but never produces exact zeros, so it keeps every feature in the model.

• Coefficients shrink smoothly
• Never exactly zero

Lambda is a hyperparameter, so we choose it with validation data, not by looking at training error. Training error always rises as lambda grows, because we are restricting the model. Validation error usually forms a U. On the left, the model has high variance. On the right, it is too constrained and has high bias. This is the bias-variance tradeoff from episode thirteen, now controlled by a single dial.

• Small lambda: high variance
• Large lambda: high bias
• Choose by cross-validation

Lasso regression replaces the squared penalty with the sum of absolute values of the weights. It looks like a small change, but the shape matters. The squared penalty is smooth and flat near zero, so tiny weights are barely penalized. The absolute value has a sharp corner at zero, and the penalty per unit of weight stays constant no matter how small the weight gets.

• Loss + lambda times sum of |w|
• Same idea, different shape

There is a second way to see the same thing. Instead of adding a penalty, imagine giving the weights a budget. Minimize the squared error, subject to the weights staying inside a region. For ridge, the region is a disk, where w one squared plus w two squared is at most t. For lasso, the region is a diamond, where the absolute values add up to at most t. A bigger lambda corresponds to a smaller budget.

• Minimize error subject to a budget
• L2 budget: circle
• L1 budget: diamond

The unconstrained best weights, the ordinary least squares solution, sit at the center of elliptical error contours. Moving away from that point, error rises, and the contours grow. The constrained solution is the first place where an expanding ellipse touches the allowed region. For the disk, that touching point is on a smooth boundary, almost always somewhere with both weights nonzero.

• Ellipses grow from the OLS point
• Solution is first touch

Now do the same with the diamond. It has sharp corners lying exactly on the axes, and at a corner one of the weights is exactly zero. Because the corners stick out, a growing ellipse usually reaches a corner before it touches a flat edge. The solution lands on the axis, and that feature is removed from the model. In higher dimensions the diamond becomes a shape with many corners and edges, so many weights can be zero at once.

• Corners lie on the axes
• Ellipses hit corners first

We can see the same effect in numbers. Suppose the features are uncorrelated and scaled so that X transposed X is the identity, and the ordinary least squares weight is z. Ridge gives z divided by one plus lambda. Lasso gives z reduced in magnitude by lambda, and clipped at zero. With lambda equal to one: if z is three, ridge gives one point five and lasso gives two. If z is point eight, ridge gives point four, but lasso gives exactly zero.

• Ridge: z over 1 plus lambda
• Lasso: shrink by lambda, clip at zero

Because lasso sets weights exactly to zero, it performs feature selection while it trains. The path plot shows features switching on one at a time as lambda decreases. The price is that, because the absolute value is not differentiable at zero, there is no tidy matrix formula. Libraries solve it with coordinate descent, which updates one weight at a time using the soft threshold we just saw.

• Sparse models
• No closed form
• Solved by coordinate descent

Lasso has weaknesses. When several features are strongly correlated, it tends to keep one and drop the others, and which one it keeps can change with small changes in the data. Also, when there are more features than rows, lasso can select at most as many features as there are rows. Ridge, by contrast, spreads weight across a correlated group.

• Picks one of a correlated group
• Unstable choice
• At most n features when p > n

Elastic net combines both penalties. Its constraint region is a diamond with rounded edges, which keeps the corners that produce zeros while adding the grouping behavior of ridge. In scikit-learn you control the overall strength and an l1 ratio between zero and one. A ratio of one is pure lasso, and a ratio near zero is nearly ridge. It is a good default when features are numerous and correlated.

• Mix L1 and L2
• Sparse and stable
• Two knobs: alpha and l1 ratio

A practical guide. If you believe many features each contribute a little, start with ridge. If you believe only a few features matter and you want a simpler model, try lasso. If you want sparsity but your features come in correlated groups, use elastic net. In every case, standardize first, and pick the strength by cross-validation.

• Ridge: many small effects
• Lasso: few important features
• Elastic net: correlated groups

Three mistakes are worth avoiding. Not scaling features, so the penalty punishes features by their units rather than their usefulness. Penalizing the intercept, which shifts the whole prediction for no good reason. And choosing lambda by looking at test error, which turns your final evaluation into just another training signal. Use cross-validation on training data, and touch the test set once.

• Forgetting to scale
• Penalizing the intercept
• Tuning on the test set

Beyond lasso, there are three families of feature selection. Filter methods score each feature on its own, for example by correlation with the target, and keep the best ones. Wrapper methods, like recursive feature elimination, repeatedly train a model and drop the weakest features. Embedded methods, like lasso, select features as part of training.

• Filter
• Wrapper
• Embedded

There is a subtle trap here. Imagine a dataset with one hundred rows and a thousand random features, and labels that are pure noise. If you select the ten features most correlated with the labels using all rows, and then cross-validate, you will get impressively good scores. That is selection leaking label information. The cure is to do selection inside each training fold, which brings us to pipelines.

• Select using training folds only
• Otherwise scores are inflated

Data leakage means that information from validation or test data influences the training process. We have already met three examples: computing scaling statistics on all the data, target encoding with all the labels, and feature selection before splitting. The result is always the same. Your estimate of performance looks better than reality, and the model disappoints in production.

• Information from outside training
• Test-time knowledge used at fit time

A scikit-learn pipeline chains preprocessing steps and a final model into one object. When you call fit, every step learns only from the data you give it. When you call predict or score on new data, the steps only transform, using what they learned during fit. Put the pipeline inside cross-validation, and each fold refits the entire chain on its own training portion.

• Chain transformers and a model
• fit learns on training only

Real datasets mix numeric and categorical columns, and each needs different treatment. A column transformer routes the numeric columns through imputation and scaling, and routes the categorical columns through imputation and one-hot encoding, then glues the results together. Setting the encoder to ignore unknown categories keeps your model from crashing when a new city shows up at prediction time.

• Numeric: impute and scale
• Categorical: impute and one-hot
• Combine columns

To tune the penalty, wrap the pipeline in a grid search or use a cross-validated ridge. In scikit-learn the regularization strength is called alpha. You try values like point zero one, point one, one, ten and one hundred. Each value is evaluated by five-fold cross-validation, and each fold refits the scaler and the model from scratch. Then you refit on the full training set with the best alpha and evaluate once on the test set.

• Pipeline plus RidgeCV or GridSearchCV
• Parameter name: ridge double underscore alpha

Here is the complete recipe. Split off a test set first and lock it away. Build a pipeline with imputation, encoding, scaling, polynomial expansion if needed, and a regularized model. Tune the hyperparameters with cross-validation on the training data only. Refit on all the training data. Then evaluate once on the test set, and report that number honestly.

• Split first
• Pipeline with preprocessing and model
• Cross-validate, then test once

Let us recap. Polynomial features give a linear model curves and interactions, at the cost of rapid growth in the number of columns. One-hot encoding gives categories their own weights without inventing an order. Standardization puts features on equal footing, and it must be computed from training data only. Correlated features make X transposed X nearly singular and the coefficients unstable.

• Polynomials add flexibility
• One-hot avoids false order
• Scale using training statistics

Ridge adds lambda times the identity before inverting, which guarantees a solution and shrinks weights smoothly. Lasso uses the absolute value penalty, whose diamond shaped constraint has corners on the axes, so it sets weights exactly to zero. Elastic net blends the two, keeping sparsity while handling correlated groups. And pipelines keep the entire process honest.

• Ridge: (X^T X + lambda I) inverse X^T y
• Lasso: sparse via diamond corners
• Elastic net: both

In the next episode, we move to a completely different way of learning: k-nearest neighbors, where a prediction comes from the closest examples, and Naive Bayes, which applies Bayes' theorem to classification. Distance and feature scaling will matter even more there.

• Next: k-Nearest Neighbors and Naive Bayes
• Subscribe for episode 18