A model scored 99% accuracy and was completely useless. The cause: data leakage.
Hook
99% Accuracy?
My model scored ninety-nine percent accuracy. It was completely useless.
- Too good to be true
Story
Scale First, Split Later
Here is what happened. I scaled the whole dataset first, then split it into train and test.
- Step 1: scale everything
- Step 2: split
The Scaler Peeked
The scaler peeked at the test rows. Their mean and spread leaked into training.
- Test statistics leak into training
The Leaked Column
Another version of the same bug: a column that quietly contained the answer, like a field filled in after the outcome.
- A feature that knows the answer
- Recorded after the outcome
Reality Hits
In production, accuracy collapsed, because the real world never hands you the answer in advance.
- Great in testing
- Collapses in production
Fix
Split First, Then Fit
The fix is one idea. Split first. Fit the scaler on training data only.
- Fit on train only
One Line: Use a Pipeline
Wrap both steps in a Pipeline. Call fit on the training set, and the test set is only ever transformed.
- make_pipeline(scaler, model)
- fit on train only
Payoff
Too Good? Suspect Leakage
If a score looks too good, suspect leakage first. Lesson thirty-five shows how to split safely. Follow for more.
- Lesson 35: safe splits