Python and Data Foundations for ML: NumPy, Pandas, EDA and Data Leakage

Here is an uncomfortable truth about machine learning. A model can score beautifully on your laptop and then fail the moment it meets real data, and the algorithm is often not the problem. The bug is hiding in the data code around it. Today we build the Python and data skills that keep you safe from that.

• Not in the algorithm
• In the code around it

Last time, we learned how loss functions measure error and how gradient descent walks downhill to reduce it. Now we need the tools to feed real data into those ideas. This episode is fast and practical, but every section connects back to the mathematics you already know.

• Last time: loss and gradient descent
• Today: tools and data hygiene

Our plan is Python basics, NumPy arrays and broadcasting, Pandas tables, cleaning, plots, feature types, outliers, and finally train, validation and test splits with a real leakage example. Our running dataset ships inside scikit-learn: the breast cancer dataset, with 569 samples, 30 numeric features, and a label of malignant or benign.

• Python, NumPy, Pandas
• Plots, outliers, splits
• Breast cancer dataset

Install Python, then install the core libraries with pip: numpy, pandas, matplotlib, seaborn, scikit-learn and jupyter. Launch a notebook and you get cells that run one at a time, with output right below each cell. A common mistake is running cells out of order, so when results look strange, restart the kernel and run everything from the top.

• Install the libraries
• Notebook cells run in order

Python variables are labels attached to values. An integer like 3, a float like 3.5, a string like malignant, and a boolean like True are the four basic types. Python is dynamically typed, so the type lives in the value, not the name. Watch out: integer division with a double slash drops the fraction, so seven double slash two gives three.

• int, float, str, bool
• Types are checked at runtime

A for loop visits items one at a time. Take the list 2, 4, 6. Start a total at zero, add each item, and you get 12. Divide by the length, 3, and the mean is 4. An if statement inside the loop lets you filter, such as counting only values above 3. Indentation is not decoration in Python, it defines the loop body.

• for loops walk through items
• if decides what to do

A list comprehension builds a list in a single readable line. Squaring each of 1, 2, 3 becomes: x squared for x in the list, giving 1, 4, 9. You can add a condition to keep only some items. This style appears everywhere in data code, and it is the stepping stone toward the vectorized thinking NumPy asks of you.

• Compact way to build lists
• Same loop, one line

A function packages a calculation. Remember mean squared error from last episode? In Python it is a function taking true values and predictions, and returning the average squared difference. With true values 3 and 5, and predictions 2 and 7, the errors are 1 and negative 2, squared they are 1 and 4, and the mean is 2.5.

• Name, inputs, return value
• Write once, reuse everywhere

A class bundles data with the functions that use it. Imagine a tiny scaler. Its fit method stores the mean from the data it sees, and its transform method subtracts that stored mean. Fit remembers, transform applies. This exact fit and transform pattern is how scikit-learn works, and it will matter a lot when we reach leakage.

• Data plus behavior
• fit then transform

Four structures cover most needs. A list is ordered and changeable. A tuple is ordered and fixed, perfect for shapes like 569 comma 30. A dictionary maps keys to values, such as a feature name to its mean. A set keeps only unique items, handy for finding the distinct classes in a column.

• List: ordered, changeable
• Tuple: fixed
• Dict: key to value
• Set: unique items

Pure Python lists can hold anything, and that flexibility costs speed and memory. Looping over millions of numbers one by one is slow. NumPy arrays store a single type in a compact block and run operations in optimized compiled code. That is why nearly every ML library is built on arrays.

• Loops are slow on big data
• Arrays store numbers compactly

An array has a shape and a data type. The array 1, 2, 3 has shape three. Stack two rows of three numbers and the shape is two comma three: two rows, three columns. Our breast cancer features form a matrix with shape 569 comma 30. Always print the shape; most bugs in ML code are shape bugs.

• shape gives dimensions
• dtype gives number type

Indexing starts at zero. For a matrix, X open bracket one comma two close bracket picks row one, column two. A colon means everything along that axis, so X open bracket colon comma zero close bracket is the entire first column. A boolean mask like X greater than five keeps only elements that satisfy the test, which is how you filter data without a loop.

• Zero-based indices
• Slices pick ranges
• Boolean masks filter

Remember the dot product from episode three? Take weights 2, 1, 3 and features 4, 5, 1. Multiply pairwise to get 8, 5 and 3, then add: 16. In NumPy that is a single call, np dot of w and x. Write the math once, and the code reads almost exactly like the formula. This is the core of every linear model we will build.

• No loop needed
• Matches the math from episode 3

The axis argument says which dimension to collapse. Take the matrix with rows 1, 2, 3 and 4, 5, 6. Summing along axis zero collapses the rows and gives one total per column: 5, 7, 9. Summing along axis one collapses the columns and gives one total per row: 6 and 15. The axis you name is the axis that disappears.

• axis zero: down the rows
• axis one: across the columns

Here is a classic trap. An array of shape three is a flat list. An array of shape three comma one is a proper column. They look similar when printed, but they behave differently in matrix operations and broadcasting. Labels and predictions are often one-dimensional, so check shapes, and use reshape with negative one comma one when you need a column.

• (3,) is not (3,1)
• reshape fixes intent

Broadcasting lets NumPy combine arrays of different shapes without copying data. Line up the shapes from the right. Each pair of sizes must be equal, or one of them must be one. A size of one is virtually stretched to match the other. If neither condition holds, NumPy raises an error, and that error is your friend.

• Compare shapes from the right
• Equal or one: compatible
• Size one stretches

A worked example. Let X have rows 1 and 10, 2 and 20, 3 and 30. The column means are 2 and 20. Subtract the mean vector from X. Shape three comma two minus shape two broadcasts the mean across every row. The result is negative 1 and negative 10, then 0 and 0, then 1 and 10. Each column is now centered at zero.

• Subtract the column means
• Shape (3,2) minus (2,)

Now divide by the standard deviation. The first column has a standard deviation of about 0.816, the second about 8.16, using the population formula. Dividing, both columns become negative 1.22, zero, and 1.22. Two features at wildly different scales now match. This is standardization, and gradient descent loves it because the loss surface becomes rounder.

• Subtract mean, divide by std
• z-scores put features on one scale

Broadcasting can also fail silently. Subtract an array of shape three from an array of shape three comma one, and you do not get three numbers. You get a three by three grid, because both arrays get stretched. No error appears, and your loss quietly becomes nonsense. The defense is simple: print shapes after every operation, and use assert statements.

• (3,) minus (3,1) gives (3,3)
• No error, wrong answer

NumPy holds numbers, but real data has column names and mixed types. Pandas gives us the DataFrame, a table with labeled rows and columns. Load the breast cancer data with as underscore frame equal to True and the shape is 569 rows by 31 columns: 30 features plus the target. Three methods tell you everything first: head for a peek, info for types and missing counts, describe for summary statistics.

• Table with named columns
• shape, head, info, describe

Square brackets with a name select a column. The iloc accessor selects by integer position and loc selects by label. Boolean masks filter rows: keep rows where mean radius is above 15. Count the target column and you find 212 malignant and 357 benign samples, so about 63 percent benign. Always check class balance early.

• df['col'] picks a column
• iloc by position, loc by label
• Masks filter rows

Group by is split, apply, combine. Split the rows by target, compute the mean of each feature in each group, and combine the results into a small table. Comparing the two groups shows which features differ between malignant and benign samples. In this data, malignant tumors tend to have larger radius measurements. That is the signal a model will try to learn.

• groupby splits by class
• Summaries per group reveal signal

A common pandas mistake is modifying a slice and wondering why the original did or did not change. When you filter a DataFrame and then assign into the result, pandas may warn you about setting a value on a copy. The safe habit is to call copy explicitly when you want an independent table, and to use loc for assignments.

• Slices may be views or copies
• Use .copy() and .loc

The breast cancer dataset is unusually tidy: it has no missing values. Real data is rarely so kind. Sensors fail, forms are left blank, and merged tables do not match. So let us practice on a tiny made-up table, clearly illustrative, with ages 25, missing, 40 and 35.

• This dataset has no gaps
• Real projects do

Pandas represents a gap as NaN, not a number. Calling isna gives a table of true and false, and chaining sum counts the gaps in each column. For our toy ages, the count is one out of four. Also watch for disguised missing values, such as an age of minus one or 999, which Pandas will happily treat as real numbers.

• isna marks gaps
• sum counts them per column

You have three options. Drop the row, which wastes information. Fill the gap with a typical value. For ages 25, 40 and 35, the mean is 33.3 and the median is 35. Or add a flag column saying the value was missing, because missingness itself can be informative. Median is safer when outliers are present, as we will see shortly.

• Drop rows: lose data
• Fill with mean or median
• Add a missing flag

Here is the first hint of leakage. If you compute that fill value using the whole dataset, test rows have already influenced your preprocessing. The correct way is to compute the mean on the training rows only, store it, and reuse that stored number for validation and test. Hold that thought, because it is exactly what the fit and transform pattern is for.

• Compute the mean on train
• Reuse it on validation and test

Exploratory data analysis means looking before modeling. Start with histograms. A histogram bins values and counts how many fall in each bin, showing center, spread and skew. Plot mean area and you will see a long tail to the right: most tumors are small and a few are large. Skewed features and strange spikes are clues about cleaning and transformations.

• Look before you model
• Skew, spread, odd spikes

A scatter plot shows two features at once. Plot mean radius against mean area, colored by diagnosis. The points form a smooth curve, which makes sense because area grows with the square of radius. And the two colors largely separate, with malignant samples toward the larger end. Already you can see that a simple model has something to learn.

• Radius vs area
• Colored by diagnosis

A correlation heatmap shows the correlation between every pair of features, from negative one to one, as a colored grid. In this dataset, many features are strongly correlated, like radius, perimeter and area, because they measure related geometry. That redundancy is called multicollinearity, and we will deal with it later. Remember, correlation shows features move together, not that one causes the other.

• Correlation from negative 1 to 1
• Seaborn heatmap
• Correlation is not causation

A box plot summarizes a feature with its median, its quartiles, and whiskers, with dots beyond them as potential outliers. Draw one box per class and you can compare the groups directly. Seaborn does this in one line. Pair it with a simple bar chart of class counts, and you have checked balance, spread and separation in just a few plots.

• Box plot: median, quartiles, whiskers
• Compare groups side by side

Features come in types, and the type decides the treatment. Numeric features can be continuous, like weight, or discrete, like a count. Categorical features are nominal, like color with no order, or ordinal, like small, medium and large with an order. Binary features have two values. Text and dates need their own handling. Identify the type of every column before doing anything else.

• Numeric: continuous or discrete
• Categorical: nominal or ordinal
• Binary, text, dates

Models need numbers, but careless encoding invents false meaning. If you map red to 1, green to 2 and blue to 3, the model may treat blue as three times red. For nominal categories, use one-hot encoding: one binary column per color. For ordinal categories, mapping small, medium, large to 1, 2, 3 is reasonable because the order is real.

• One-hot for nominal
• Order mapping for ordinal
• Avoid fake order

An outlier is a value far from the rest. A standard rule uses the interquartile range. Take the values 10, 12, 13, 14, 15, 16, 18 and 60. The first quartile is 12.75 and the third is 16.5, so the range is 3.75. The upper fence is 16.5 plus 1.5 times 3.75, which is 22.125. Sixty lies far beyond it, so it is flagged.

• IQR equals Q3 minus Q1
• Fences: 1.5 times IQR beyond quartiles

Do not delete outliers automatically. First ask whether it is an error, like a typo, or a real rare event, which may be exactly what you need to predict. See how much one value matters: the mean of those eight numbers is 19.75, but the median is 14.5. Squared losses punish extreme values heavily, so outliers can drag a model toward them.

• Investigate first
• Mean moves, median resists

Now the most important idea of the episode. A model that memorizes its training data looks perfect on that data and tells us nothing about new data. So we split. The training set is where parameters are learned. The validation set is where we compare models and tune choices. The test set is the final exam, touched once at the very end.

• Training: learn
• Validation: tune
• Test: final exam

With 569 samples, hold out 20 percent for test: scikit-learn rounds up to 114 rows, leaving 455. Take 25 percent of those, 114 rows, for validation. That leaves 341 for training. Use random state for reproducibility, and stratify on the label, so each split keeps roughly the same 63 percent benign share instead of drifting by chance.

• Test 20 percent: 114 rows
• Validation 25 percent of rest: 114
• Train: 341

Here is a concrete leak. Suppose training values are 10, 20, 30 and the single test value is 100. Training mean is 20. But if you standardize before splitting, the mean uses all four values and becomes 40. The test point has pulled the scaling, so training features now carry information about data the model should never have seen. The model looks better in evaluation than it truly is.

• Mean computed on all rows
• Test info enters training

Leakage has other forms. Target leakage happens when a feature is only known after the outcome, like a treatment code assigned after diagnosis, so the model reads the answer. Duplicate rows landing in both train and test let the model memorize. And with time series, random splitting lets the model peek at the future. Ask of every feature: would I truly have this at prediction time?

• Features recorded after the outcome
• Duplicate rows across splits
• Time order broken

The safe pattern is simple. Split first. Fit the scaler on the training set only. Then transform training, validation and test with those stored numbers. A scikit-learn Pipeline bundles the scaler and the model so this happens automatically, even inside cross validation. When your score looks suspiciously perfect, suspect leakage before celebrating.

• fit on train only
• transform train, validation, test
• Pipelines enforce it

One more rule. Every time you look at test results and change something, the test set quietly becomes part of training. Make decisions on validation, then evaluate on test once, and report that number honestly, even if it disappoints you. A modest honest score beats an impressive fake one, because honest scores predict how the model behaves in the real world.

• Use validation for decisions
• Test once, at the end

Let us recap. Python gives us functions and classes, and the fit and transform pattern. NumPy gives fast arrays, where shapes and broadcasting rules decide everything. Pandas holds labeled tables, and cleaning means handling missing values and outliers thoughtfully. Plots reveal structure. And above all, split first, fit on training data only, and keep the test set sacred.

• Python, NumPy, Pandas
• Plots, types, outliers
• Split first, fit on train

Next time, in episode 13, we use these splits to ask the central question of machine learning: why does a model that fits the training data perfectly sometimes fail on new data? We will meet underfitting, overfitting and the bias variance trade-off.

• Underfitting and overfitting
• Bias-variance trade-off