Two Full ML Projects: House Prices and Customer Churn, Start to Finish

Anyone can fit a model in three lines of code. What separates a tutorial from real work is everything around those three lines. Today we build two complete projects: predicting house prices, and predicting which customers will leave. Both follow exactly the same template, and the differences between them will teach you more than either one alone.

• House prices
• Customer churn
• Same workflow

In the last lesson, we covered MLOps: reproducible pipelines, serving, deployment and monitoring. Now we put the whole course to work. We will lean on exploratory analysis from lesson 33, cleaning and splits from lessons 34 and 35, residual analysis from lesson 40, boosting from lesson 42, and tuning from lesson 45.

• Lesson 63: MLOps
• Capstone begins now

Here is the template. One, define the problem and the metric. Two, explore the data. Three, clean it and split it safely. Four, build a baseline. Five, fit a linear or regularized model. Six, fit gradient boosting. Seven, tune. Eight, evaluate on the test set once. Nine, analyse errors and state limitations honestly.

• Define, explore, clean, split
• Baseline, model, boost, tune
• Evaluate, analyse, admit limits

A baseline is the dumbest sensible model. For regression, predict the median price for every house. For classification, predict that nobody churns. If your fancy model cannot clearly beat that, you have learned something important before spending any more time on it.

• Baseline sets the bar
• Beat it, or stop

One rule for this whole lesson: report only numbers from your own runs. I will use tiny made-up examples to show the arithmetic, but I will not quote scores for you to copy. Run the code, keep your random seeds and library versions, and trust your own output. The test set gets touched exactly once, at the end.

• Report only your own runs
• Test set used once

Our first project uses a public housing dataset, such as the Ames housing data. The target is the sale price in dollars, a continuous number, so this is regression. Features include living area, overall quality, year built, and neighborhood. Before touching data, decide who uses the prediction, because that decides which error matters.

• Target: sale price
• Features: size, quality, age, location
• Metric decided up front

Exploration, as in lesson 33, starts with the target. Plot a histogram of prices and you will see a long right tail: many modest homes, a few very expensive ones. Then plot the histogram of the natural log of price. The tail pulls in and the shape looks far closer to a bell curve.

• Prices are right-skewed
• A few huge outliers
• Log makes it symmetric

Why does this help? A two hundred thousand dollar house and a four hundred thousand dollar house differ by a factor of two. The log of that ratio is about zero point six nine, the same gap as between one hundred thousand and two hundred thousand. So an error in log space is a percentage error, which matches how people think about prices.

• Errors become percentages
• Taming the tail
• Prices multiply, logs add

Now cleaning and splitting, from lessons 34 and 35. Split into train and test first. Then put imputation, scaling and one-hot encoding inside a pipeline, so medians and category lists are learned from training rows only. Missing values in a feature like garage area may mean there is no garage, so think before filling with a median.

• Split first
• Fit imputers on train only
• Use a pipeline

First the baseline: predict the training median for every house. Next, ridge regression from lesson 17, which adds a penalty on squared weights so correlated features such as living area and room count do not produce wild coefficients. Standardize numeric features so the penalty treats them fairly, and choose the penalty strength with cross-validation on the training set.

• Median baseline
• Ridge regression
• Standardize features

Linear models assume each feature acts independently and additively. Gradient boosting, from lesson 42, builds trees one after another, each fitted to the remaining residuals of the ensemble so far. That lets it learn interactions, like how an extra bedroom matters differently in a small house than in a big one. A small learning rate and many trees usually works well.

• Trees fit residuals in turn
• Captures interactions
• Shrinkage via learning rate

For tuning, lesson 45 gave us random search and Bayesian optimization. Search over a handful of knobs: number of trees, learning rate, tree depth, and minimum leaf size. Score every candidate by cross-validation inside the training set. If you tune against the test set, even informally, your final score is no longer an honest estimate.

• Random or Bayesian search
• Cross-validation inside train
• Few, important knobs

Take four houses with errors of ten, ten, ten and fifty thousand dollars. The mean absolute error is the average: eighty over four, so twenty thousand. For root mean squared error, square in thousands to get one hundred, one hundred, one hundred and twenty-five hundred, average to seven hundred, and take the square root: about twenty-six and a half thousand. RMSE is larger because it punishes the big miss.

• Errors: 10k, 10k, 10k, 50k
• MAE = 20k
• RMSE about 26.5k

Because we trained on the log target, RMSE in log space is roughly a typical percentage error. An RMSE of zero point one in logs means predictions are typically off by around ten percent. Also report MAE in real dollars for stakeholders. Remember that exponentiating a log prediction gives something closer to a median than a mean, so a small bias in dollars is possible.

• RMSE on log price
• Roughly a percentage error
• Report dollars too

Lesson 40 taught us to read residuals. Plot residuals against predictions: you want a shapeless cloud around zero. A curve means missing nonlinearity. A funnel, spreading as predictions grow, means non-constant variance, which the log target often fixes. A Q-Q plot reveals heavy tails, and a handful of extreme points may deserve a closer look.

• Residuals vs predictions
• Look for curves and funnels
• Q-Q plot for tails

Now sort the test rows by absolute error and read the worst twenty. Are they unusual luxury homes, odd sales, or a neighborhood with few examples? Also state limits plainly: the data covers one city and one period, so market shifts will degrade the model, and a price predictor is no substitute for an appraisal.

• Where are the biggest misses?
• Rare luxury homes
• Market drift over time

Project two uses a public telecom-style churn dataset. Each row is a customer with contract type, tenure, monthly charges and services, and the label says whether they left. The goal is to rank customers by risk so a retention team can contact the right ones. Notice that the real objective is an action, not a prediction.

• Target: left or stayed
• Telecom-style public data
• Imbalanced classes

Suppose a thousand customers, of whom a hundred churn. A model that predicts everyone stays is right nine hundred times, which is ninety percent accuracy, yet it finds zero churners. This is why lesson 16 insisted on precision and recall. Here those matter far more than accuracy.

• 1000 customers, 100 churn
• Predict nobody leaves
• 90 percent accuracy, zero use

Use a stratified split so train and test keep the same churn rate. Then hunt for leakage. A column such as cancellation reason or last month's total charges recorded after leaving would let the model peek at the answer. Ask of every feature: would I know this on the day I make the prediction? If not, drop it.

• Stratify on the label
• Beware post-churn columns
• Fit encoders on train only

The baseline is the majority class, with precision-recall scores that make its uselessness obvious. Next comes logistic regression from lesson 41, with an L2 penalty and one-hot encoded categories. You can use class weights to emphasise the rare class, but be aware this shifts the predicted probabilities upward, which will matter when we reach calibration.

• Majority-class baseline
• Regularized logistic regression
• Class weights optional

Next fit gradient boosting and tune it as before. To compare models on imbalanced data, draw the precision-recall curve: precision on the vertical axis, recall on the horizontal, one point per threshold. A curve hugging the top right is better, and the area summary, average precision, should be compared against the base rate, which is what a random ranker earns.

• Boosted trees for tabular data
• Precision-recall curve
• Compare average precision

Here is a made-up business case. A retention offer costs twenty dollars. If a churner accepts, which happens forty percent of the time, we keep three hundred dollars of value. Expected gross gain from contacting a customer with churn probability p is zero point four times three hundred times p, or one hundred twenty p. Contact only if that beats twenty.

• Offer costs 20
• Saved churner worth 300
• 40 percent accept

Solve one hundred twenty p greater than twenty, and p must exceed one sixth, about zero point one seven. That is nowhere near the default of zero point five. With a cost-driven threshold we contact more people, accepting more false alarms because a miss costs more than a wasted offer. If costs change, the threshold moves, and you never need to retrain.

• 120p > 20
• p > 1/6, about 0.17
• Not 0.5!

That cost rule only works if probabilities are honest. A calibrated model that says thirty percent should be right about thirty out of every hundred such customers. Plot a reliability diagram: predicted probability against observed churn rate, and compare to the diagonal. If the curve bends away, fit Platt scaling or isotonic regression on validation data, never on the test set.

• Say 30 percent, mean 30 percent
• Reliability diagram
• Fix on validation data

Finally, interpretation with SHAP from lesson 20. For one customer, SHAP shows how each feature pushes the risk up or down from the average, such as a month-to-month contract pushing up and long tenure pushing down. Averaged across customers it ranks global importance. Treat it as explaining the model, not the world; it shows association, not causes.

• Per-customer explanations
• Global importance
• Association, not causation

Look at the churners the model missed. Do they share a segment, or leave for reasons the data never captured, like a competitor's promotion? A deeper limitation: predicting who will leave is not the same as predicting who will be persuaded to stay. Contacting customers who would have stayed anyway wastes money, and that question needs experiments.

• Who is missed?
• Drift as offers change
• Prediction is not uplift

Put the projects side by side. In housing, the output is a number, so we measure error size, and the log target reflects that percentage mistakes matter. In churn, the output is a ranking turned into an action, so we measure precision, recall, expected cost, and calibration. Same template, different metrics, because the decision each model supports is different.

• House: RMSE, MAE, log scale
• Churn: PR curve, cost, calibration
• Metric follows the decision

Let us collect the classic mistakes. Fitting scalers or imputers before splitting. Tuning on the test set. Celebrating accuracy on imbalanced data. Using the default threshold of zero point five. Trusting raw probabilities from a weighted model. And skipping the baseline, which leaves you unable to say whether the model helped at all.

• Preprocessing before splitting
• Tuning on the test set
• Default 0.5 threshold
• Uncalibrated probabilities

Connecting to lesson 63, save the whole fitted pipeline as one artifact, not just the model. Record random seeds, library versions, data snapshot and the metric values from your run. Write a one-page report covering the problem, the baseline, the final score, the threshold logic and the limitations. Future you, and your teammates, will be grateful.

• Seeds and versions
• Saved pipeline
• One-page model report

To recap: both projects followed one template, from problem definition through baseline, linear model, boosting, tuning, evaluation and honest limits. House prices taught us log targets, RMSE versus MAE, and residual diagnostics. Churn taught us the precision-recall curve, a cost-based threshold, calibration and SHAP. Always choose the metric from the decision.

• One template, two problems
• Log target and residuals
• Cost threshold and calibration

We deliberately left out fairness-sensitive data today. In the next lesson, capstones three and four, we tackle credit-risk classification with fairness checks, and then anomaly detection. See you in lesson 65.

• Lesson 65
• Fairness-aware classification
• Anomaly detection