Data Cleaning for ML: Missing Values, Outliers and Feature Types

Imagine training a model to predict salaries, and one person in your table earns two hundred and fifty thousand while everyone else earns around forty. Another row has an age of minus three. A third is just blank. The model will not complain. It will quietly learn from all of it, and give you confident nonsense.

• Models trust their data
• Dirty data, wrong answers

In lesson thirty three, exploratory data analysis taught us to look at the data with plots and summaries. Today we act on what we found. We will classify feature types, handle missing values, and deal with outliers, all with real numbers and scikit-learn code.

• Lesson 33: EDA reveals problems
• Lesson 34: fix them

Here is the plan. First, what kinds of columns exist, because the type decides what cleaning even makes sense. Then the three flavors of missing data and how to fill them. Then outliers: how to detect them, tame them, and when to leave them alone. Encoding and scaling belong to lesson seventeen, so we will not repeat them here.

• Feature types
• Missing data
• Outliers
• scikit-learn code

Every column belongs to a type. Numeric columns hold quantities, like price or height. Ordinal columns have an order but uneven gaps, like small, medium, large. Nominal columns are unordered categories, like city. Then there are datetimes, free text, and identifiers.

• Numeric, ordinal, nominal
• Datetime, text, IDs

Numeric splits further. Continuous values, like temperature, can land anywhere on a number line. Discrete values, like the number of children, are counts. The distinction matters because averages of counts can be fractions, and a mean of two point three children is a summary, not a person.

• Continuous: any value in a range
• Discrete: counts

Ordinal versus nominal is where beginners slip. Education level, low, medium and high, has a real order, but the gap between low and medium is not necessarily the gap between medium and high. A city name has no order at all, so Paris is not greater than Rome. Mislabeling a nominal column as numeric invents an order that does not exist.

• Ordinal: order matters
• Nominal: no order

A datetime is not a number you should feed raw. Extract useful parts like month, weekday or days since signup. Free text needs special processing, which we meet later. And an ID column, such as a customer number, usually carries no real signal. A model can memorize it, which is just overfitting in disguise.

• Datetime: extract parts
• Text: needs its own treatment
• ID: usually drop

Now the quality problems EDA reveals. First, duplicates. If the same customer appears twice, the model counts them twice, and worse, one copy may land in training and the other in testing. Your test score then looks better than reality. Check with a duplicate count, then drop exact repeats after confirming they are errors.

• Same row twice
• Inflates some patterns
• Leaks across splits

Second, inconsistent labels. A city column may contain New York, new york, NY, and New York with a trailing space. To a computer these are four different categories. Strip whitespace, lowercase everything, and map known variants to one canonical name. Then count the unique values again and watch the category list shrink.

• NY, ny, New York
• Trailing spaces
• Merge to one label

Third, impossible values. An age of minus three, a height of nine hundred centimeters, a birth date in the future. These are not unusual, they are wrong. Write simple rules from domain knowledge, such as age between zero and one hundred twenty, and flag every row that breaks them. Often the fix is to treat the bad value as missing.

• Age of -3
• Height of 900 cm
• Check against domain rules

Sometimes missing values hide in plain sight. Old systems use minus nine nine nine, zero, or the word unknown to mean no data. If you leave them, the mean gets dragged to nonsense. Look for suspicious spikes in histograms, and convert these placeholders to real not-a-number markers before doing anything else.

• -999, 0, 'unknown'
• Convert to real NaN

Before filling any blank, ask why it is blank. The reason decides whether a simple fix is safe or quietly harmful. Statisticians describe three mechanisms: missing completely at random, missing at random, and missing not at random. The names are confusing, so let us make each one concrete.

• The reason matters
• Three mechanisms

Missing completely at random, or MCAR, means the blank has nothing to do with any value, seen or unseen. A sensor battery dies at random moments, or a clerk spills coffee on a few forms. The missing rows are a fair sample of all rows, so deleting or filling them does not bias the averages, though you do lose information.

• Missing completely at random
• Unrelated to any value
• Safest case

Missing at random, or MAR, means the chance of a blank depends on other columns you can see. Suppose younger people skip the income question more often. Income is missing because of age, not because of income itself. Here, the observed columns carry information we can use to fill the gap sensibly, which is why model-based imputation helps.

• Missing at random
• Depends on observed data
• Example: age and income

Missing not at random, or MNAR, is the hard one. The blank depends on the very value that is missing. High earners refuse to state their income, or very sick patients skip the follow-up visit. No column fully explains it. Simple imputation will be biased, and the best you can do is add flags, collect better data, or model the missingness.

• Missing not at random
• Depends on the hidden value
• Hardest case

You cannot prove which mechanism holds from the data alone, but you can collect clues. Count missing values per column, then compare other columns between rows with and without a blank. If the average age differs sharply between the two groups, it is not MCAR. Treat that as evidence, then reason about how the data was collected.

• Count missing per column
• Compare groups with and without
• Missingness heatmap

The simplest fix is deletion. Drop every row with a blank, or drop a column that is mostly empty. With ten thousand rows and only fifty blanks, that is fine under MCAR. But if each of ten columns is missing five percent independently, about forty percent of rows contain at least one blank, so you could throw away nearly half your data.

• Drop rows (listwise)
• Drop columns
• Costs data, risks bias

Next is simple imputation. Replace blanks in a numeric column with the mean or median, and in a categorical column with the most frequent value, the mode. Take incomes thirty, thirty two, thirty five, thirty eight, forty, forty one, forty five and two hundred fifty. The mean is sixty three point eight, but the median is thirty nine. The median is a far more honest fill when outliers exist.

• Numeric: mean or median
• Categorical: mode
• Simple and fast

Mean imputation has a hidden cost. Take ages twenty, thirty, forty and fifty, with a mean of thirty five. The squared deviations sum to five hundred, and divided by four, the variance is one hundred twenty five. Now fill two blanks with thirty five. Those new points add zero deviation, so we divide five hundred by six and get eighty three point three. We invented data and the spread collapsed.

• Filled values sit at the center
• Spread artificially drops

Why does shrinking the variance matter? Because every filled point sits exactly at the center, relationships between columns weaken. A scatter plot that once showed a clear slope now has a flat row of identical values across it. Correlations get pulled toward zero, and later statistical tests become overconfident, because the data looks tighter than it really was.

• Weakens correlations
• Too-confident models

When missingness itself is informative, keep that information. Add a new binary column, income was missing, equal to one if the original was blank and zero otherwise. Then fill the original column with the median. The model gets a reasonable number and also a clear signal. Under MNAR, this flag can capture exactly what the blank was hiding.

• Add a 0/1 column
• Keeps the 'was missing' signal

A smarter fill uses similar rows. For kNN imputation, find the k nearest neighbors using the columns that are present, then average their values for the missing column. Suppose a person aged thirty two has no income. Their two nearest neighbors, by age and years of experience, earn forty and forty four. With k equal to two, we fill in the average, forty two.

• Find similar rows
• Average their values

So which to choose? With very few blanks under MCAR, deletion or median is fine. When missingness depends on other columns, kNN or model-based imputation does better. When a blank may carry meaning, add a flag. Whatever you choose, compare validation performance across strategies instead of trusting taste.

• Few blanks, MCAR: delete or median
• MAR: kNN or model-based
• Always consider a flag

Now outliers, points far from the bulk of the data. They come from two very different sources. Some are mistakes, like a typo that added an extra zero. Others are real but rare, like a billionaire in an income survey. Detecting a far point is easy. Deciding what it means takes judgment.

• A point far from the rest
• Error, or real and rare?

The z-score measures how many standard deviations a value lies from the mean: z equals x minus the mean, divided by the standard deviation. A common rule flags absolute values above three. Using our incomes, the mean is sixty three point nine and the standard deviation is seventy point five. For two hundred fifty, z is one hundred eighty six divided by seventy point five, about two point six four.

• z = (x - mean) / std
• Flag if |z| > 3

Notice something troubling. Two hundred fifty is obviously extreme, yet its z-score is only two point six four, below the cutoff of three. The outlier itself inflated the mean and the standard deviation, hiding its own strangeness. In fact, with eight points the biggest possible z-score is about two point six five, so the three sigma rule can never fire here.

• Outlier inflates mean and std
• z only 2.64, under 3
• With 8 points, max z is about 2.65

A sturdier method is the interquartile range rule. Sort the incomes, take the median of the lower half for Q1, thirty three point five, and of the upper half for Q3, forty three. The IQR is nine point five. Multiply by one point five to get fourteen point two five. The fences are nineteen point two five and fifty seven point two five. Two hundred fifty is far outside, so it is flagged.

• Q1, Q3, IQR = Q3 - Q1
• Fences: Q1 - 1.5 IQR, Q3 + 1.5 IQR

The IQR rule works because quartiles are robust: a few extreme values barely move them. Compare the mean, sixty three point nine, with the median, thirty nine. Remove the outlier and the mean drops to thirty seven point two, yet the median only moves to thirty eight. Whenever outliers are possible, summarize with the median and the IQR.

• Median and IQR resist outliers
• Mean and std do not

Once you decide an extreme value should not dominate, you can cap it instead of deleting the row. Clipping replaces anything above the upper fence with the fence itself. Winsorizing does the same using percentiles, like the first and ninety ninth. Clip our two hundred fifty to fifty seven point two five, and the mean falls from sixty three point nine to about thirty nine point eight.

• Cap at the fences
• Keep the row, tame the value

Here is the crucial warning. Sometimes the outlier is the whole point. A credit card transaction ten times larger than usual may be fraud. A machine vibration spike may predict failure. If you delete or clip these, you erase what you were hired to find. Always investigate: trace the source, ask a domain expert, and document your decision.

• Fraud, failures, rare events
• Never delete blindly

A good workflow has four steps. Detect with the IQR rule or plots. Investigate whether it is an error or real. If it is an error, correct it or mark it missing. If it is real, keep it, cap it, or use a model robust to extremes, such as a tree based method. Never let a rule make the decision for you.

• Detect, investigate, then decide
• Fix error, cap, keep, or model separately

Now the code. In scikit-learn, SimpleImputer takes a strategy argument: mean, median, most frequent, or constant. You call fit on the data to learn the fill value, then transform to apply it. Set add indicator to true and it appends the missing-indicator columns for you, so the flag and the fill happen in one step.

• strategy: mean, median, most_frequent
• fit then transform

For kNN imputation, use KNNImputer with n neighbors set to two. It measures distances using the columns that are present and averages the neighbors' values, just as in our forty and forty four example. Because it relies on distances, columns with big scales can dominate, so scaling matters here, a topic covered in lesson seventeen.

• n_neighbors=2
• Uses distances on present columns

For outliers, compute the first and third quartiles with numpy percentile, build the fences from the IQR, and call numpy clip to cap values between them. Do this per column, and keep a boolean mask so you can inspect which rows changed. Logging what you modified is part of honest data cleaning.

• Quantiles with numpy
• np.clip to the fences

One rule overrides everything. Every statistic we compute, the median to fill with, the fences to clip at, must be learned from the training data only, then applied unchanged to validation and test data. Computing them on the full dataset leaks information from the future into your model. We will build leakage safe splits and pipelines properly in lesson thirty five.

• Medians, means, fences: learn from train
• Apply the same values to test
• Details in lesson 35

Let us review the classic mistakes. Imputing before splitting the data. Treating a nominal column as numeric. Trusting the three sigma rule on small samples. Deleting outliers that were actually fraud. And filling blanks without ever asking why they were blank. Each one is easy to avoid once you know to look for it.

• Imputing before splitting
• Deleting signal outliers
• Ignoring why data is missing

Recap. Identify each column's type and fix duplicates, inconsistent labels and impossible values. Understand whether data is missing completely at random, at random, or not at random, then choose deletion, median or mode, flags, or kNN, remembering that imputation shrinks variance. Detect outliers with the IQR rule, prefer robust statistics, cap carefully, and keep outliers that are signal.

• Types, then quality checks
• Missing: MCAR, MAR, MNAR
• Outliers: IQR, robust, judgment

In the next lesson, we put this into practice safely: leakage safe train, validation and test splits, and pipelines that guarantee your cleaning is fitted only on training data. Thanks for watching, and I will see you in lesson thirty five.

• Lesson 35 next
• Subscribe to continue