Why Your 99% Model Is a Lie: Leakage-Safe Splits and Pipelines

Imagine you train a model and it scores ninety nine percent on the test set. You celebrate, you ship it, and in production it barely beats a coin flip. Nothing is wrong with the algorithm. The number was a lie, and the cause is almost always a single word: leakage.

• Test accuracy: 99%
• Production accuracy: far lower

Leakage means the model saw information during training or preprocessing that it will not have when it meets truly new data. The test score then measures something easier than the real task. In this lesson we name the four main kinds of leakage, learn how to split data safely, and build pipelines that make leakage hard to commit.

• Information from outside training
• Sneaks into the model
• Inflates the estimate

Last time, in lesson 34, we cleaned missing values, outliers and feature types. Today we decide when that cleaning is allowed to happen. We go through leakage types, then splitting, then pipelines, then a head to head comparison, and finally a checklist you can keep. Group and time series cross validation get only a preview here.

• Four kinds of leakage
• Safe splits
• Pipelines
• Audit checklist

The first kind is target leakage. A feature secretly contains the answer, or is created after the answer is known. Suppose you predict whether a patient has a disease, and one column records whether they received the disease specific treatment. Treatment follows diagnosis, so the model just reads the answer off that column.

• A feature that encodes the answer
• Unavailable at prediction time

Another example: predicting loan default, with a feature counting collections calls. Collections calls only happen after someone stops paying. At the moment you must make the prediction, at application time, that column does not exist yet. The question to ask of every feature is simple: will I truly know this value at the moment of prediction?

• Predict loan default
• Column: collections calls made
• Happens only after default

Target leakage often shows up as a single feature with enormous importance, or a model that is near perfect on a messy real world problem. Treat that as an alarm, not a victory. Trace where each column comes from and when it is recorded. If a column is produced by the outcome, drop it, however useful it looks.

• Suspiciously strong feature
• Top importance, near perfect
• Ask: when is it recorded?

The second kind is subtler. Preprocessing leakage happens when you scale, impute, encode, or select features using the whole dataset before splitting. The mean, the median, the category list are all learned from data. If those learned numbers include test rows, then information from the test set has quietly flowed into training.

• Fit statistics on all rows
• Test data influences training

Let us use numbers. Training values are two, four and six, and the test value is twenty. Fit on training only, the mean is four, and the population standard deviation is the square root of eight thirds, about one point six three. The test value becomes sixteen divided by one point six three, roughly nine point eight. The model sees the test point is wildly unusual.

• Train values: 2, 4, 6
• Test value: 20
• Fit on train only

Now do it the leaky way and fit on all four values. The mean jumps from four to eight, because the test point dragged it. The spread grows too, so the twenty looks less extreme than it really is. The scaler has used the test point to describe itself, and any later evaluation is quietly flattering the model.

• Fit on all four values
• Mean becomes 8
• Outlier looks less extreme

The same trap appears everywhere. A median imputer fit on all rows leaks. Choosing the top features by correlation with the target, using all rows, leaks badly, because the selection has already looked at test labels. Target encoding of categories leaks the same way. Any step that learns from data must learn from training data only.

• Median fit on all rows leaks
• Feature selection on all rows leaks
• Target encoding leaks

The cure is a clear habit. Fit learns numbers from the training data. Transform applies those stored numbers to any data. So you fit the scaler on the training set, then transform training, validation and test with that same fitted scaler. Never call fit, or fit transform, on validation or test data.

• fit learns from train
• transform applies the learned numbers
• Never fit on test

The third kind is duplicate leakage. If the same row, or a near copy, appears in both training and test sets, the model can simply remember it. Scraped data, merged tables and repeated exports often produce duplicates. Remove exact duplicates before splitting, and check near duplicates too, because a memorized answer looks exactly like real skill.

• Same row in train and test
• Memorization looks like skill

Group leakage is duplicate leakage in disguise. Say each of one thousand patients has five scans, giving five thousand images. A random row split puts some scans of a patient in training and others in testing. The model learns that patient's anatomy, not the disease. The fix is to split by patient, so every group lands entirely on one side.

• Rows belong to a group
• Patient, user, device, house
• Keep each group on one side

The fourth kind is temporal leakage. When the task is to predict the future, a random shuffle lets the model train on events after the ones it is tested on. It can learn patterns that were only visible in hindsight. Instead, train on the past and test on the later period, exactly as the model will be used.

• Predicting the future
• Random split mixes past and future
• Train on past, test on future

Temporal care extends to features. A rolling average must be computed from the past only, and a lag feature must really be a lag. If a centered window peeks at future days, you have built a time machine into your inputs. Scalers and imputers for time data must also be fit only on the earlier period.

• Lag features only from the past
• Rolling means must exclude future
• Fit scalers on the past

There are cross validation methods built for these cases: group based folds that keep each group together, and time ordered folds that always test on the future. We will not teach them today. Lesson 24 covers cross validation in depth. For now, remember that the right splitter depends on how your data was generated.

• GroupKFold keeps groups together
• TimeSeriesSplit respects order
• Taught in lesson 24

Now let us split properly. We want three sets. The training set fits the parameters. The validation set is where we compare models and tune hyperparameters. The test set is a sealed envelope, used for one final honest grade. Using test data to make choices turns it into a second validation set, and the honesty is gone.

• Train: learn parameters
• Validation: choose models
• Test: final honest grade

In scikit learn, train test split takes your features and labels, a test size, and a random state. With ten thousand rows and test size zero point two, you get eight thousand training rows and two thousand test rows. The random state is the seed. Fix it, and anyone rerunning your code gets the identical split.

• test_size sets the fraction
• random_state fixes the shuffle
• Same seed, same split

When classes are imbalanced, add stratify equals y. Suppose one thousand rows, with one hundred positives, ten percent. A twenty percent test split has two hundred rows. A plain random split might give fourteen positives, or twenty seven, which makes metrics jumpy. Stratified splitting keeps the ten percent ratio, so the test set gets twenty positives.

• 1000 rows, 100 positives
• Test 200 rows
• Stratified: exactly 20 positives

For three sets, call the splitter twice. First carve off twenty percent as test: eight thousand rows remain, two thousand are locked away. Then split the eight thousand with a test size of zero point two five, because a quarter of eight thousand is two thousand. You end with six thousand, two thousand, two thousand. Keep stratify and the seed in both calls.

• Split 1: carve off test 20%
• Split 2: validation is 25% of rest
• Result: 60 / 20 / 20

Here is the rule: the test set is touched once. Make the split at the very beginning, lock it away, and look at it only after every modeling decision is final. If the score disappoints you, resist tuning against it. Each peek and adjustment leaks a little information back, and after enough rounds your test score is just another training signal.

• Lock it away early
• Evaluate once, at the end
• Bad score? Do not retune on it

Let us collect the usual mistakes. Preprocessing before splitting. Forgetting the seed, so results cannot be reproduced. Shuffling data that has a time order. Splitting rows when the real unit is a patient or a user. And the quiet one: checking the test score many times and calling it validation. All of these are avoidable with structure.

• Splitting after preprocessing
• Forgetting the seed
• Shuffling time data
• Tuning on the test set

Remembering to fit only on training data is fragile. A pipeline makes it automatic. In scikit learn, Pipeline chains steps, such as an imputer, a scaler and a model, into one object. Calling fit runs each step on the training data in order and stores what was learned. Calling predict pushes new data through the same stored steps.

• Chain steps into one estimator
• fit runs every step on train
• predict reuses learned numbers

Pipelines matter most inside cross validation. Pass the whole pipeline to the cross validation function, and for each of the five folds the scaler and imputer are refit on that fold's four training parts only. The held out part stays untouched. If you scaled first and then cross validated, every fold would have been influenced by its own validation data.

• Five folds, five refits
• Each fold: fit on 4 parts
• Validate on the held-out part

Real tables mix types, so we use a column transformer. Send the numeric columns through a median imputer and a scaler. Send the categorical columns through a most frequent imputer and a one hot encoder. The results are joined side by side into one matrix. Then wrap this preprocessor and a model together in an outer pipeline.

• Numeric columns: impute, scale
• Categorical columns: impute, one-hot
• Combine side by side

One practical detail. The one hot encoder learns its category list from training data. If a new city appears at test time, the default behavior is an error. Setting handle unknown to ignore encodes it as all zeros instead. This is the right behavior, because the model should cope with the unfamiliar, not have its list quietly extended by test rows.

• New category at test time
• handle_unknown ignore
• Encoder learns categories from train

Hyperparameter search plugs in the same way. Give a search object the whole pipeline, and name parameters as step name, a double underscore, then the parameter. You can even tune the imputer strategy. Every candidate is evaluated with preprocessing refit inside each fold, so the search itself stays leakage safe.

• Search over pipeline parameters
• step name, double underscore, parameter
• Preprocessing refit in every fold

Let us run a clean comparison on the same data. Take a dataset with many noisy features and labels that are pure random coin flips, so no model can truly do better than chance. In the leaky workflow we select the best features and scale using all rows, then split. In the clean workflow we split first and use a pipeline.

• Same data, two workflows
• Leaky: select and scale first
• Clean: pipeline after split

Among hundreds of random features, some will correlate with the labels by pure luck, and selection on all rows finds exactly those, test rows included. The leaky workflow then reports accuracy well above fifty percent on data with no real signal at all. Nothing real was learned. The estimate was manufactured by peeking.

• Feature selection saw test labels
• Accuracy well above 50%
• Pure noise looks like signal

The clean workflow selects features on the training rows only, then scores on test rows it never touched. The accuracy lands near fifty percent, which is the truthful answer: there is nothing here to learn. The bad looking number is the good news. It stopped you from shipping a model built on noise.

• Selection fit on train only
• Accuracy near 50%
• Honest answer: no signal

On real data the gap between workflows varies. With plenty of rows and few features, scaling leakage may change the score only a little. With small data and many features, selection leakage can be dramatic, as we just saw. You cannot know in advance, so the safe default is always to build the clean version.

• Leakage gap is often smaller
• Smaller data, more features: bigger gap
• Never trust it will be small

Here is a checklist you can run on any project. One: does every feature exist at the moment of prediction? Two: is the split unit correct, rows, groups, or time? Three: were exact and near duplicates removed before splitting? These three questions catch target, group and duplicate leakage before any code runs.

• Does each feature exist at prediction time?
• Is the unit of splitting correct?
• Were duplicates removed first?

Four: split before any step that learns from data, including imputing, scaling, encoding and feature selection. Five: put every such step inside a pipeline, so fitting is confined to training folds. Six: fix seeds and save the split indices, so results are reproducible and nobody can quietly resplit.

• Split before any learning step
• Every transform inside a pipeline
• Fixed seeds, saved splits

Seven: use the test set once. Eight: if a result seems too good, investigate before celebrating. Compare against a trivial baseline, like predicting the majority class, and inspect feature importances for a suspicious leader. The best habit is a skeptical one. Strong results deserve more checking, not less.

• Test set used exactly once
• Suspiciously good? Investigate
• Compare to a trivial baseline

Let us recap. Leakage inflates scores by letting outside information into training. Watch for target leakage, preprocessing leakage, duplicate and group leakage, and temporal leakage. Split first, stratify when classes are imbalanced, fix your seed, and touch the test set once. Use pipelines and column transformers so every transform is fit on training data only.

• Four leaks: target, preprocessing, group, temporal
• Split first, stratify, seed, lock test
• Pipelines enforce fit on train only

That finishes the data foundations for this course. Next, in lesson 36, we return to mathematics with the chain rule and computational graphs, the machinery behind backpropagation.

• Lesson 36: Chain rule
• Computational graphs
• Subscribe for the full course