Random Forests, Boosting and SHAP Explained With Real Numbers

Imagine asking one person to guess the weight of an ox at a fair. They will probably be off by a lot. Now ask eight hundred people and average the guesses. The average is often startlingly close. Today we will turn that idea into some of the most successful algorithms in machine learning.

• One guess: noisy
• Average of many: stable

In the previous episode we built decision trees, which split the feature space with simple questions. We saw that a deep tree fits training data perfectly but wobbles wildly when the data changes. That wobble is high variance. Today we turn that weakness into a strength by combining many trees.

• Deep trees: low bias
• But high variance

Here is the plan. First bootstrap sampling and bagging, then random forests and out of bag scoring. Next, boosting: AdaBoost and gradient boosting, derived carefully. Then XGBoost, LightGBM and CatBoost, stacking and voting, and finally feature importance and SHAP, so you can explain what these powerful models are actually doing.

• Bagging and forests
• Boosting
• Stacking
• SHAP

We only have one dataset, but we want many different training sets. The trick is the bootstrap: draw n rows from our n rows, with replacement. Some rows get picked twice or three times, and some are never picked. Every bootstrap sample is the same size as the original, yet slightly different.

• Draw n rows with replacement
• Duplicates allowed
• Some rows left out

Take ten rows numbered one to ten. A bootstrap draw might give us three, seven, seven, one, nine, three, three, ten, five, and two. Row three appears three times. Rows four, six and eight never appear. Those three missing rows are called out of bag for this sample, and they will become useful later as a free test set.

• Sample: 3,7,7,1,9,3,3,10,5,2
• Out of bag: 4, 6, 8

How many rows are typically left out? The chance a specific row is missed on one draw is one minus one over n. Missing it on all n draws gives that quantity to the power n. For n equals ten that is zero point nine to the tenth, about zero point three four nine. As n grows this approaches one over e, about zero point three six eight.

• P(missed) = (1 - 1/n)^n
• n = 10: 0.349
• n large: 1/e ≈ 0.368

Flip that around. Roughly sixty three point two percent of the original rows appear at least once in a bootstrap sample, and about thirty six point eight percent do not. A common mistake is to think the sample contains only sixty three percent as many rows. It still has n rows, but many are duplicates.

• ~63.2% appear
• ~36.8% out of bag
• Sample size is still n

Bagging means training one model per bootstrap sample, then combining them. For classification we take a majority vote. For regression we average the predictions. Each tree sees slightly different data, so each makes slightly different mistakes, and averaging smooths those mistakes out.

• Train B trees on B samples
• Classify: majority vote
• Regress: average

Suppose each model has prediction variance sigma squared, and the models are independent. The variance of their average is sigma squared divided by B. With a variance of four and one hundred models, the averaged variance is just zero point zero four. Averaging does not change bias, but it crushes variance.

• Var(average) = σ² / B
• σ² = 4, B = 100 → 0.04

Our trees are not independent, because they train on overlapping data. If each pair has correlation rho, the variance of the average becomes rho times sigma squared, plus one minus rho times sigma squared over B. With rho equals zero point five, sigma squared four, and B one hundred, that is two plus zero point zero two, so two point zero two.

• Var = ρσ² + (1−ρ)σ²/B
• ρ = 0.5 → 2.02
• Correlation sets a floor

This formula tells us something deep. Adding more trees only shrinks the second term. The floor, rho times sigma squared, stays. To go lower we must make the trees less alike. That insight is exactly what motivates random forests: inject randomness so the trees disagree more.

• More trees: diminishing returns
• Lower ρ: lower floor

A random forest is bagging with one extra twist. At every split, each tree may only consider a random subset of the features. Common defaults are the square root of the feature count for classification, and about a third of the features for regression. Strong features can no longer dominate every tree, so trees become more diverse.

• Bootstrap rows
• Random feature subset per split
• √p for classification

Say we have two hundred trees voting on whether a loan defaults. One hundred forty trees say yes and sixty say no. The forest predicts yes, and the fraction of votes, zero point seven, serves as a rough probability. Be careful though: vote fractions are often poorly calibrated, a topic from our metrics episode.

• 200 trees
• 140 yes, 60 no
• Vote share = 0.70

Remember the rows each tree never saw. For every row, collect predictions only from trees where it was out of bag, and combine them. Compare with the true label across all rows and you get the out of bag score. It behaves like cross validation without refitting anything, which makes it almost free.

• Each row: use trees that skipped it
• Combine, compare to truth
• Free validation estimate

Take a row that was out of bag for sixty trees. Forty two of those trees predict class one and eighteen predict class zero. The out of bag prediction is class one. Repeat for all rows, and if ninety of one hundred rows are right, the out of bag accuracy is ninety percent.

• 60 OOB trees
• 42 vote class 1
• OOB accuracy = 90/100

The main knobs are the number of trees, the features per split, and tree depth or leaf size. More trees rarely hurt, they just cost time. A common mistake is reporting the training score of a forest, which is almost perfect and meaningless. Use out of bag or a proper validation set instead.

• n_estimators: more is safe
• max_features: key knob
• Never trust training score

Bagging trains models in parallel to reduce variance. Boosting trains models in sequence, where each new model focuses on what the previous ones got wrong. It uses weak learners, such as very shallow trees, and slowly builds a strong one. Boosting mainly attacks bias.

• Sequential, not parallel
• Weak learners
• Targets bias

AdaBoost gives every training row a weight, starting equal at one over n. A weak learner is trained with those weights, so heavy rows matter more. Rows it misclassifies get heavier, and rows it gets right get lighter. The next learner is forced to pay attention to the hard cases.

• Start weights = 1/n
• Mistakes gain weight
• Next learner focuses there

Use ten rows, each with weight zero point one. The first stump misclassifies three, so its weighted error is zero point three. Its say, called alpha, is one half times the natural log of the ratio of one minus error over error. That is one half times log of seven over three, about zero point four two four.

• ε = 0.3
• α = ½ ln((1−ε)/ε)
• α ≈ 0.424

Multiply each misclassified weight by e to the alpha, about one point five two eight, giving zero point one five two eight. Multiply each correct weight by e to the minus alpha, about zero point six five four, giving zero point zero six five four. These sum to zero point nine one six two, so we divide by that total to normalize.

• Wrong: 0.1 × 1.528 = 0.1528
• Right: 0.1 × 0.654 = 0.0654
• Normalize by 0.9162

After normalizing, each misclassified row has weight about zero point one six seven, which is one sixth, and each correct row about zero point zero seven one, one fourteenth. Check: three sixths plus seven fourteenths is one half plus one half. The mistakes now carry exactly half the total weight, so the old stump is no better than a coin flip on the new problem.

• Wrong rows: 1/6 each
• Right rows: 1/14 each
• Mistakes hold exactly 50%

The final model is a weighted vote: the sign of the sum of alpha times each learner's output, where outputs are plus or minus one. If three stumps have alphas of zero point four, zero point seven and zero point two, and they vote plus, minus, plus, the total is zero point four minus zero point seven plus zero point two, which is minus zero point one. The ensemble says minus. AdaBoost is also sensitive to label noise, since noisy rows keep gaining weight.

• F(x) = sign(Σ αₘ hₘ(x))
• 0.4 − 0.7 + 0.2 = −0.1
• Sensitive to noisy labels

AdaBoost is a clever recipe, but gradient boosting gives a general principle. We want a function F that minimizes a loss over the data. Instead of adjusting a parameter vector, we adjust the function itself, adding a small correction at each step. This is gradient descent in function space.

• Goal: minimize Σ L(yᵢ, F(xᵢ))
• Update the function, not weights

Treat the predictions F of x i, one per row, as the variables. For squared error, the loss for a row is one half times y minus F, squared. Its derivative with respect to F is minus the quantity y minus F. So the negative gradient is exactly y minus F, which is the residual. Stepping down the gradient means adding the residual.

• L = ½(y − F)²
• ∂L/∂F = −(y − F)
• Negative gradient = residual

Start with a constant prediction, the mean for squared error. Then repeat: compute residuals, fit a small tree to predict them, and update F by adding a learning rate times that tree. The tree is needed because we can only change F at the training points by a function that generalizes to new inputs. The tree is our approximation of the gradient.

• F₀ = mean(y)
• Fit tree to residuals
• Fₘ = Fₘ₋₁ + η·hₘ

Let targets be ten, twenty and thirty. The starting prediction is the mean, twenty. Residuals are minus ten, zero and ten. Suppose a tree fits them perfectly and the learning rate is zero point one. New predictions are nineteen, twenty and twenty one, with residuals of minus nine, zero and nine. Each round shrinks the residuals by ten percent.

• y = 10, 20, 30; F₀ = 20
• Residuals: −10, 0, +10
• η = 0.1 → F₁ = 19, 20, 21

The learning rate trades off against the number of trees. Small steps need many trees but generalize better, because each tree contributes only a little. With a rate of one, early trees dominate and overfit quickly. A common practice is a small rate like zero point zero five to zero point one, with early stopping on a validation set to choose the tree count.

• Small η: more trees, smoother
• Large η: fast but overfits
• Use early stopping

The power of this view is that any differentiable loss works. For logistic loss, the negative gradient is y minus the predicted probability p. For absolute error, it is the sign of the residual, which makes the model robust to outliers. We only change what the trees are fitted to; the loop stays the same.

• Log loss: residual = y − p
• Absolute error: sign of residual
• Same loop, new gradient

XGBoost uses both the gradient g and the second derivative h of the loss, and adds a penalty on tree complexity. For a leaf, the best output is minus the sum of g divided by the sum of h plus lambda. If the gradients sum to minus six, hessians sum to three, and lambda is one, the leaf value is six over four, or one point five. Lambda shrinks leaf values and fights overfitting.

• w* = −G / (H + λ)
• G = −6, H = 3, λ = 1
• w* = 1.5

LightGBM speeds up training by bucketing feature values into histograms, so split searching touches a few hundred bins instead of every unique value. It also grows trees leaf-wise, always splitting the leaf with the largest loss reduction, rather than level by level. This is fast and accurate, but on small datasets leaf-wise growth can overfit, so limit the number of leaves.

• Histogram binning
• Leaf-wise growth
• Cap num_leaves on small data

CatBoost is designed to handle categorical columns directly. Naively replacing a category by its average target leaks the label into the feature, which is a form of data leakage. CatBoost uses ordered statistics, computing each row's encoding only from rows that came before it in a random permutation. It also uses symmetric trees, which are fast and regularized.

• Native categorical handling
• Ordered target statistics
• Symmetric trees

At a practical level, all three are excellent, and differences are usually smaller than the effect of good features and tuning. Choose LightGBM for large data and speed, CatBoost when you have many categorical columns, and XGBoost as a robust, well documented default. Always compare on your own validation split rather than trusting folklore.

• LightGBM: large, fast
• CatBoost: categoricals
• XGBoost: solid default

Voting combines different model types, such as logistic regression, a forest, and k nearest neighbors. Hard voting takes the majority class. Soft voting averages predicted probabilities, which is usually better because confident models count for more. If the models give zero point nine, zero point four and zero point four, soft voting averages zero point five six seven and predicts class one.

• Hard: majority class
• Soft: average probabilities
• 0.9, 0.4, 0.4 → 0.567

Stacking goes further: a meta model learns how to weigh the base models. The crucial detail is that the meta model must train on out of fold predictions, made by base models that did not see those rows. If you feed in predictions on training data the base models already memorized, the meta model learns to trust the overfit ones.

• Base models → predictions
• Meta-model learns weights
• Use out-of-fold predictions

Pull the ideas together. Bagging and forests average low bias, high variance trees, so variance falls. Boosting adds shallow, high bias trees sequentially, so bias falls. Stacking and voting exploit different models making different errors. Diversity is the common ingredient, and it is why identical models gain nothing from being combined.

• Bagging: lowers variance
• Boosting: lowers bias
• Diversity is essential

Forests report importance as the total impurity decrease contributed by each feature across splits. It is quick, but biased toward features with many unique values, such as an ID column, and it is computed on training data. When two features are highly correlated, importance gets split between them, making each look less important than it is.

• Biased to high-cardinality
• Computed on training data
• Correlated features split credit

A better check is permutation importance. Take a validation set, shuffle one column, and measure how much the score drops. If accuracy falls from ninety to eighty four percent, that feature's importance is six points. It reflects real held-out performance, though correlated features can still mask each other.

• Shuffle one column
• Measure score drop
• 90% → 84% = 6 points

Both methods tell us which features matter globally. They do not say whether a feature pushes predictions up or down, nor why a single person was denied a loan. For that we want local explanations, where each prediction is split into contributions from each feature. This is where Shapley values enter.

• Global: which matter overall
• Local: why this prediction
• Direction up or down

Shapley values come from game theory: how to divide a payout fairly among players who cooperate. Features are the players and the prediction is the payout. The value of a feature is its average marginal contribution over all possible orders in which features could join the coalition.

• Features = players
• Prediction = payout
• Average over all orderings

The average prediction is fifty. With only feature A known, the prediction is sixty five. With only B, it is fifty eight. With both, eighty. For A: joining first adds fifteen, joining after B adds twenty two, average eighteen point five. For B: first adds eight, after A adds fifteen, average eleven point five. Together they sum to thirty, exactly eighty minus fifty.

• A: (15 + 22) / 2 = 18.5
• B: (8 + 15) / 2 = 11.5
• 18.5 + 11.5 = 30 = 80 − 50

SHAP applies Shapley values to machine learning predictions, guaranteeing that contributions plus the base value equal the model output. Computing them naively needs exponentially many coalitions. TreeSHAP exploits tree structure to compute exact values for forests and boosted trees in practical time. Plots include waterfall charts for one prediction, and summary plots across the dataset.

• Contributions sum to prediction
• TreeSHAP: fast and exact for trees
• Waterfall and summary plots

A few cautions. SHAP explains the model, not the real world, so it describes what the model learned, not causes. Correlated features can share credit in surprising ways, and the choice of background data changes the baseline. Treat SHAP as a powerful diagnostic that must be combined with domain knowledge.

• Explains model, not causality
• Correlation shares credit
• Background data matters

In scikit-learn, set up a classification dataset and a five fold cross validation. Compare a single decision tree, a random forest with three hundred trees, gradient boosting, and optionally XGBoost, LightGBM and CatBoost, using the same splits. On typical tabular data you will see the single tree trail well behind, with forest and boosting models clustered near the top.

• Same splits for all
• Tree < forest ≈ boosting
• Report mean and spread

With a forest, set oob score to true and read the attribute afterward. For boosting, pass a validation set and an early stopping round count, so training halts when the validation loss stops improving. Finally, create a TreeExplainer, compute SHAP values on a held-out sample, and plot a summary. Always fit preprocessing inside a pipeline so no information leaks.

• oob_score=True
• early_stopping_rounds
• shap.TreeExplainer

Let us recap. Bootstrap samples leave out about thirty seven percent of rows. Bagging averages models to cut variance, and random forests decorrelate trees with random features. Boosting fits residuals, which are negative gradients, one small tree at a time. Stacking combines diverse models, and SHAP splits a prediction fairly among features.

• Bagging and forests: variance
• Boosting: bias
• SHAP: fair credit

Next time we leave trees behind and meet support vector machines. We will find the widest possible margin between classes, use hinge loss, and see how kernels let straight line methods draw curved boundaries.

• Next: SVMs and kernels
• Subscribe for episode 21