Model Evaluation and Selection Done Right: CV, Tuning, Leakage

Your model just scored ninety-five percent accuracy. You are thrilled, you deploy it, and a month later it performs far worse than promised. Nothing broke. The score was simply never trustworthy in the first place.

• 95 percent in the notebook
• Much worse in production

Last time, in episode twenty-three, we reduced dimensions with PCA, t-SNE and UMAP. Today we ask a harder question: how do we measure a model honestly, and choose between models without fooling ourselves? This is the skill that separates careful practitioners from lucky ones.

• Previously: dimensionality reduction
• Today: trustworthy measurement

Here is our roadmap. We will cover cross-validation and its special variants, hyperparameter search, learning curves, statistical uncertainty, leakage, nested cross-validation, error analysis and reproducibility. Every idea comes with concrete numbers and scikit-learn code.

• Splits and search
• Curves and uncertainty
• Leakage, nested CV, tracking

The simplest evaluation is a single train-test split. Suppose you hold out twenty percent of your data and get eighty-five percent accuracy. The question is: if you had shuffled differently, would you get eighty-five again?

• 80 percent train, 20 percent test
• One number, one shuffle

Let us try it. Run the same split with ten different random seeds, and you might see accuracies like eighty-one, eighty-eight, eighty-four, and ninety. The model never changed. Only the luck of which rows landed in the test set changed.

• Ten seeds, ten answers
• Spread comes from the split

This spread is variance in our estimate, not variance in the model. A small test set makes it worse, because a few unusual rows can swing the result. If you then pick the model with the best single split, you are partly picking the luckiest split.

• Small test sets swing wildly
• Picking the best split picks luck

There is a second problem. When we tune a model while peeking at the test score, the test set quietly becomes part of training. The standard fix is three roles: train to fit, validation to choose, and test to report once at the very end.

• Train fits parameters
• Validation chooses models
• Test is opened once

Cross-validation reduces the luck. In k-fold cross-validation, we cut the data into k equal folds. We train on k minus one folds, validate on the remaining one, and rotate until every fold has been the validation fold exactly once.

• Split into k folds
• Rotate the validation fold
• Every row validated once

Let us do a real calculation with five folds. The five validation accuracies are zero point eight two, zero point eight five, zero point eight zero, zero point eight four and zero point eight four. Their sum is four point one five, so the mean is zero point eight three.

• Scores: .82 .85 .80 .84 .84
• Sum 4.15, mean 0.83

The mean alone hides the spread. The deviations from zero point eight three are minus zero point zero one, plus zero point zero two, minus zero point zero three, plus zero point zero one and plus zero point zero one. Squared, they sum to zero point zero zero one six. Dividing by four gives a variance of zero point zero zero zero four, so the standard deviation is zero point zero two.

• Squared deviations sum to 0.0016
• Variance 0.0004, std 0.02

So we report eighty-three percent, plus or minus two points across folds. That standard deviation describes how much fold scores vary. The uncertainty in the mean itself is smaller, roughly the standard deviation divided by the square root of five, about zero point zero zero nine, though folds share training data so this is only a rough guide.

• 0.83 plus or minus 0.02
• Folds are not fully independent

In scikit-learn this is one line. You call cross val score with your estimator, your features and labels, and cv equals five. It returns an array of five scores, and you take the mean and standard deviation. Under the hood, each fold gets a fresh clone of the model, so nothing carries over.

• cross_val_score(model, X, y, cv=5)
• Fresh model clone per fold

How many folds should you use? Five or ten is the common default. More folds mean more training data per fit, so less pessimistic estimates, but more compute. At the extreme, leave-one-out uses n folds, which is expensive and often gives high-variance estimates.

• Five or ten is typical
• More folds, more compute
• Leave-one-out is rarely needed

A common mistake is fitting a scaler on the whole dataset before cross-validation. The scaler then learns the mean and spread of the validation rows, which leaks information. The cure is a Pipeline, which refits the scaler inside each training fold only.

• Fit scaler on train fold only
• Use a Pipeline

Plain k-fold assumes rows are interchangeable. Real data often breaks that assumption: classes are imbalanced, rows come in groups, or time has an order. For each case, scikit-learn offers a splitter that respects the structure.

• Imbalanced classes
• Grouped rows
• Ordered in time

Take a fraud dataset with one thousand rows and fifty positives, so five percent. With five random folds, each fold holds two hundred rows and should contain ten positives. But a random shuffle could easily give one fold four positives and another sixteen, making fold scores wildly inconsistent.

• 1000 rows, 50 positives
• Ideal: 10 positives per fold
• Random: 4 to 16

Stratified k-fold fixes this by splitting each class separately, so every fold keeps the original class proportions. Each fold then holds exactly ten positives and one hundred ninety negatives. In scikit-learn, passing an integer cv to a classifier already uses stratification by default, but it is good to know it explicitly.

• Split each class separately
• Same proportions in every fold

Now groups. Imagine a medical imaging dataset with one hundred patients and ten scans each, so one thousand images. If scans of the same patient land in both train and validation, the model can recognize the patient rather than the disease. Validation scores look excellent and mean nothing.

• 100 patients, 10 scans each
• Same patient in train and validation
• Inflated scores

Group k-fold solves this by keeping all rows of a group in the same fold. With five folds, each fold holds twenty whole patients. You pass the group labels to the split call, as in cross val score with groups equal to patient ids. The honest question is always: will new data come from new groups?

• All rows of a group stay together
• Pass groups to the splitter

Time is the third case. If you forecast sales, you cannot train on next year and validate on last year, because the future would inform the past. Time series split uses expanding windows: train on month one, test on month two; train on months one and two, test on month three; and so on.

• Train only on the past
• Expanding training window
• Never shuffle time

Now to choosing hyperparameters. Parameters are learned from data, like weights. Hyperparameters, like the regularization strength C or the tree depth, are set before training. We pick them by comparing cross-validation scores across candidates.

• Parameters: learned
• Hyperparameters: chosen
• Chosen by validation score

Grid search tries every combination. Suppose C has four values and gamma has five. That is twenty combinations, and with five folds each, one hundred model fits. In scikit-learn, GridSearchCV takes a parameter grid, runs all of them, and exposes best params and best score.

• 4 values of C, 5 of gamma
• 20 combinations, 100 fits

Grids scale badly. With five hyperparameters of five values each, you face three thousand one hundred twenty-five combinations. Random search samples configurations from distributions instead. Research by Bergstra and Bengio argued that random search is often more efficient, because usually only a few hyperparameters matter much.

• Grid: 3 distinct values per axis
• Random: 9 distinct values
• Better coverage of what matters

In code, RandomizedSearchCV takes distributions, like a log-uniform range for C, plus n iter for the budget. Learning rates and regularization strengths should be sampled on a log scale, because the difference between zero point zero zero one and zero point zero one matters as much as between zero point one and one.

• Use log-uniform for C and rates
• n_iter sets the budget

Bayesian optimization is smarter. It builds a surrogate model of score as a function of hyperparameters, then picks the next trial where the model expects improvement, balancing exploring unknown regions against exploiting good ones. Libraries such as Optuna and scikit-optimize implement this idea, and it shines when each fit is expensive.

• Surrogate model of the score
• Balance explore and exploit
• Good for expensive fits

One warning: the best score from a search is optimistically biased. If you try hundreds of configurations, the winner is partly the one that got lucky on the folds. Keep this in mind, because we will fix it shortly with nested cross-validation.

• Best of many is biased upward
• Search score is not final score

Before tuning blindly, diagnose. A validation curve plots training and validation score against one hyperparameter, such as tree depth. At low depth, both scores are low: underfitting. At high depth, training score is near perfect while validation drops: overfitting.

• Score versus one hyperparameter
• Low depth: underfit
• High depth: overfit

A learning curve instead varies the training set size. If training and validation scores converge at a low level, the model is too simple, and more data will not help. If there is a large gap that narrows as data grows, more data or stronger regularization may help.

• Score versus training size
• Converge low: need a richer model
• Big gap: need data or regularization

Here is a practical example. Training accuracy is ninety-nine percent and validation accuracy is eighty percent. The nineteen-point gap signals variance. Try limiting depth, adding regularization or collecting more data, and re-plot. In scikit-learn, learning curve and validation curve return the arrays you need.

• 99 train, 80 validation
• 19-point gap means variance
• Try the remedies, re-plot

Suppose model A scores eighty-five percent and model B scores eighty-six on a test set of two hundred rows. Is B better? Accuracy is an estimate, and every estimate carries uncertainty. A responsible report includes an interval, not just a point.

• Test set of 200 rows
• Scores are estimates
• Report intervals

A quick approximation: with one hundred seventy correct out of two hundred, accuracy is zero point eight five. The standard error is the square root of zero point eight five times zero point one five, divided by two hundred, which is about zero point zero two five. A ninety-five percent interval is roughly plus or minus zero point zero four nine, so eighty and ninety percent.

• 170 of 200 correct
• Standard error about 0.025
• 95 percent: 0.80 to 0.90

The bootstrap gives a more general tool that works for any metric, such as F1 or AUC. Resample your test predictions with replacement, same size as the original, compute the metric, and repeat a thousand times. The middle ninety-five percent of those values, from the two point five to ninety-seven point five percentiles, is your interval.

• Resample with replacement
• Repeat 1000 times
• Take the 2.5 and 97.5 percentiles

With a 0.80 to 0.90 interval, the one-point gap between models A and B is noise. Better yet, compare them on the same resampled rows and bootstrap the difference. If the interval for the difference comfortably includes zero, you have no evidence that one is better, and you should prefer the simpler or cheaper model.

• Bootstrap the paired difference
• Interval includes zero: no clear winner
• Prefer simpler

Leakage is when information from outside the legitimate training signal sneaks in. We saw preprocessing leakage and group leakage. The nastiest form is target leakage: a feature that encodes the answer, or is only known after the outcome happens.

• Features that encode the answer
• Information from the future

Imagine predicting loan default and including a column called collections status. It is only filled in after a customer defaults. The model scores near perfect in validation and is useless at decision time. Ask for every feature: would I truly have this value at prediction time?

• Would I know this at prediction time?
• Suspiciously perfect scores are a clue

Other leaks include duplicate rows across splits, feature selection done before splitting, and tuning against the test set. A useful red flag is a feature with absurdly high importance. When results look too good, assume a leak first and a breakthrough last.

• Duplicates across splits
• Selection before splitting
• Tuning on the test set

Now back to the winner's curse. If the same folds both choose hyperparameters and report performance, the estimate is optimistic. Nested cross-validation uses two loops. The inner loop tunes hyperparameters, and the outer loop evaluates the entire tuning procedure on data it never touched.

• Inner loop: tune
• Outer loop: evaluate the procedure

The cost is real. With five outer folds, three inner folds and, say, twenty configurations, that is five times three times twenty, or three hundred inner fits, plus five refits. In scikit-learn, you wrap a GridSearchCV object inside cross val score. The outer score estimates how well your whole pipeline, tuning included, will generalize.

• 5 x 3 x 20 = 300 inner fits
• GridSearchCV inside cross_val_score

A single metric tells you how often you are wrong, not why. Error analysis means looking at the mistakes. Pull up the misclassified rows, group them by feature values, class, or data source, and compute the metric per slice. Often one segment, like short texts or a particular camera, carries most of the errors.

• Inspect the mistakes
• Slice metrics by segment
• Find where it fails

Error analysis tells you what to do next. If errors cluster in a mislabelled segment, fix the labels. If they cluster where data is scarce, collect more there. If they are spread evenly, change the model or features. A good diagnosis beats a hundred blind tuning runs.

• Noisy labels: fix them
• Sparse regions: add data
• Spread evenly: change the model

Finally, reproducibility. Set random state in your splitters and models so a colleague can rerun your experiment and get the same numbers. But remember that one seed is one draw. Repeating with several seeds, and reporting the spread, is more honest than showing one lucky run.

• Set random_state everywhere
• One seed is one draw
• Report spread over seeds

Track everything. Record the data version, the code commit, the library versions, the hyperparameters, the split strategy and the metrics for every run. Tools like MLflow or Weights and Biases make this easy, but even a well-kept table works. If you cannot reproduce last month's best result, you do not really have a result.

• Data, code, params, split, metrics
• MLflow or a simple table

Let us recap. A single split is noisy, so use cross-validation and report mean and spread. Match the splitter to the data: stratified for imbalance, group for shared entities, time series for ordered data. Keep all preprocessing inside a pipeline.

• Cross-validate, report spread
• Match the splitter to the data
• Pipelines prevent leakage

For selection, use random or Bayesian search on sensible scales, read validation and learning curves, and quantify uncertainty with bootstrap intervals. Guard against leakage by asking what is known at prediction time. Use nested cross-validation for an honest final estimate, then analyze errors and track every run.

• Search wisely, read the curves
• Intervals, not point scores
• Nested CV, error analysis, tracking

In the next episode, we tackle time series and forecasting, where order matters and the splits you just learned become essential.

• Episode 25: forecasting
• Subscribe for the series