Trustworthy ML: Uncertainty, Shift, Fairness and Robustness Explained

Your model scores ninety five percent on the test set. Do you trust it? Not yet. Will it know when it is guessing? Will it survive next year's data? Can you explain its decisions, treat groups fairly, and resist someone trying to fool it? That is trustworthy machine learning.

• High accuracy is not trust
• Four questions to ask

In the last lesson we met Bayesian inference and Gaussian processes. Today we build a practical toolkit. We quantify uncertainty with ensembles and conformal prediction, detect shift and anomalies, explain models, reason about causes, measure fairness, and finally attack a model with a single line of algebra.

• Uncertainty and conformal
• Shift and anomalies
• Explain, cause, fairness, attack

There are two kinds of uncertainty. Aleatoric uncertainty is noise in the world itself, like predicting a fair coin flip; more data never helps. Epistemic uncertainty is ignorance of the model, because it has seen too little data in some region. Epistemic uncertainty shrinks with more data. Aleatoric does not.

• Aleatoric: irreducible noise
• Epistemic: lack of knowledge

The distinction tells you what to do. If uncertainty is aleatoric, you hedge and accept a limit on accuracy. If it is epistemic, you can collect more data in that region, or refuse to predict and ask a human. A model that cannot tell these apart will be confidently wrong exactly where it knows least.

• Aleatoric: accept or hedge
• Epistemic: collect more data

A simple way to estimate epistemic uncertainty is an ensemble. Train five models with different seeds or data samples. Suppose they predict ten point two, nine point eight, ten point five, ten point one, and nine point nine. The mean is ten point one, and the standard deviation is about zero point two five. Wide disagreement signals the model is unsure.

• Train several models
• Spread of predictions is uncertainty

Training many networks is expensive. Monte Carlo dropout is a cheaper trick. Keep dropout switched on at prediction time and run the same input through the network, say fifty times. Each pass uses a different random sub-network, so you get fifty predictions. Their mean is the answer and their variance is a rough uncertainty. It is an approximation, not exact Bayesian inference.

• Keep dropout on at test time
• Average T stochastic passes

In lesson 16 we calibrated probabilities so that among predictions of eighty percent, about eighty percent are correct. That is useful, but it is an average statement, and it relies on the model's probability being trustworthy. For regression, or for a guarantee we can state precisely, we want something stronger. Enter conformal prediction.

• Lesson 16: probabilities match frequencies
• No guarantee for one prediction

Conformal prediction wraps any model and outputs an interval, or a set of labels, that contains the true answer with a chosen probability, say ninety percent. It makes no assumption about the model being correct, and no assumption about the data distribution. It only needs the calibration and test points to be exchangeable, which roughly means drawn the same way.

• Prediction set with 1 − α coverage
• Works with any model
• Needs exchangeable data

Split conformal has four steps. First, split your data into a training set and a separate calibration set, and fit the model on training data only. Second, on each calibration point, compute a nonconformity score, which for regression is the absolute error between the true value and the prediction. Large score means the model was surprised.

• Split data: train and calibration
• Score = |y − prediction|

Step three is picking a quantile. Say we have nineteen calibration scores and want ninety percent coverage, so alpha is zero point one. Compute n plus one, which is twenty, times zero point nine, giving eighteen. Sort the scores and take the eighteenth smallest as q hat. Notice the plus one: it is a small finite sample correction that makes the guarantee exact.

• n = 19, α = 0.1
• Rank = ceil((n+1)(1−α)) = 18
• q̂ = 18th smallest score

Suppose that eighteenth score is four point two. A new house arrives, and our model predicts fifty two, in thousands. Step four: the interval is the prediction minus q hat to the prediction plus q hat. That gives forty seven point eight to fifty six point two. We claim the true price falls inside with at least ninety percent probability.

• q̂ = 4.2
• Prediction 52
• Interval [47.8, 56.2]

Why does it work? If the new point is exchangeable with the nineteen calibration points, its score is equally likely to land at any rank among all twenty. The chance it ranks within the lowest eighteen is eighteen over twenty, which is ninety percent. So coverage is at least ninety percent, and at most about ninety five here, since one over n plus one is five percent.

• Test score is one more exchangeable score
• Rank is uniform among n+1
• Coverage ≥ 1 − α

Two cautions. The guarantee is marginal: ninety percent on average over inputs, not for every individual input, so some subgroups may be undercovered. And it fails if the test data is not exchangeable with calibration data, which brings us to distribution shift. A better model does not change the coverage, but it shrinks the intervals, making them more useful.

• Guarantee is marginal, not per-input
• Breaks if data shifts
• Good models give tighter intervals

Every guarantee so far assumed that the future looks like the past. In practice, a model trained in January meets customers in December, new cameras, new laws. This is distribution shift. It helps to name three kinds, because the fixes differ: covariate shift, label shift, and concept shift.

• Training P differs from deployment Q
• Three flavours of shift

In covariate shift, the inputs change but the relationship from inputs to labels stays the same. A skin cancer model trained on young patients is deployed to older ones. The rule linking image to diagnosis is unchanged, yet the model sees regions of input space where it had little data, so epistemic uncertainty rises and errors grow.

• P(x) changes
• P(y | x) stays the same

In label shift the class balance changes while each class looks the same. Imagine fraud is five percent of training transactions but ten percent in deployment. The importance weight for fraud is ten over five, so two. For legitimate transactions it is ninety over ninety five, about zero point nine five. Reweighting training examples, or adjusting the decision threshold, corrects for the new balance.

• P(y) changes, P(x | y) fixed
• Weight w(y) = Q(y) / P(y)

Concept shift is the hardest. The very meaning of the label changes. The same email text was fine last year but is now a phishing pattern; a pandemic changes what a normal spending pattern means. Inputs may look identical, so you cannot spot it from inputs alone. You need fresh labels, and usually you need to retrain.

• P(y | x) itself changes
• Same inputs, new answers

To detect shift, train a classifier to tell training data from recent production data. If it cannot beat chance, an AUC near zero point five, the distributions look alike. If it reaches an AUC of zero point nine, they differ a lot, and its feature importances show where. Simpler options are the Kolmogorov Smirnov test or the population stability index on each feature.

• Classifier two-sample test
• KS test and PSI per feature
• Monitor performance when labels arrive

Sometimes the right response to unusual data is to flag it. Anomaly detection looks for points unlike the rest: fraud, failing machines, intrusions. Labels are rare or missing, so we use unsupervised ideas. Two popular ones are isolating points by random cuts, and measuring how badly a compressed copy reconstructs the original.

• Rare, unlabelled, unexpected
• Anomaly ≠ error

Isolation forest cuts the data with random vertical and horizontal lines. A lonely point is separated after very few cuts, while a point deep inside a crowd needs many. Average the path length h across many random trees, and the score is two to the power of minus the average path over c of n, where c of n is the typical path length. Scores near one are anomalies; near zero point five, normal.

• Random splits isolate outliers fast
• Short path = suspicious
• Score near 1 = anomaly, 0.5 = normal

The reconstruction idea trains an autoencoder, or uses PCA from lesson 23, to compress normal data and rebuild it. Normal inputs rebuild well. An unfamiliar input does not, so its reconstruction error is large. Choose a threshold from validation data, for example the ninety ninth percentile of normal errors, and flag anything above it.

• Autoencoder learns normal data
• High error = unfamiliar input

Labels cost money, but unlabelled data is abundant. In semi-supervised learning, train on say one hundred labelled examples, label the ten thousand unlabelled ones, and keep only confident predictions as pseudo labels. In self-supervised learning the data labels itself: hide a word or an image patch and predict it. That is how the language models of lesson 57 are pretrained.

• Pseudo-labels from confident predictions
• Self-supervision: predict hidden parts

Now interpretability, extending SHAP from lesson 20. Permutation importance is wonderfully simple. Measure test accuracy, say zero point nine zero. Shuffle the income column, breaking its link to the target, and accuracy falls to zero point seven two: importance eighteen points. Shuffle zip code and accuracy is zero point eight nine, so importance one point. Income matters; zip code barely does.

• Shuffle one feature
• Measure the score drop

Partial dependence shows how predictions change with one feature. Pick a grid of values for age. For each value, set every row's age to that value, predict, and average. Plot the averages. You see the model's average response curve, say risk rising steadily until sixty then flattening. Beware: if features are correlated, you create unrealistic rows, like age ten with a mortgage.

• Fix feature at value v
• Average predictions over data
• Plot against v

LIME explains a single prediction. Take one instance, generate many perturbed copies nearby, and ask the black box to predict each. Weight copies by closeness to the original, then fit a simple linear model to them. Its coefficients say which features pushed this prediction up or down locally. Like any local approximation, it can be unstable, so check it across random seeds.

• Perturb around one instance
• Weight by closeness
• Fit a sparse linear model

A common mistake is to read an explanation as a statement about reality. These tools explain what the model uses, not what causes the outcome. If the model leans on hospital ward to predict pneumonia risk, that is a shortcut, not a medical cause. Use several methods, and when they disagree, investigate rather than choose the story you like.

• Explain the model, not the world
• Importance is not causation

Prediction asks what will happen. Causal inference asks what happens if we intervene. Ice cream sales correlate with drowning, but hot weather drives both. Weather is a confounder. In data, patients who receive a drug may be sicker, or healthier, than those who do not, so comparing their outcomes mixes the drug's effect with the patients' differences.

• A hidden variable drives both
• Correlation ≠ causation

The gold standard is the randomized experiment. Flip a coin to decide who gets the treatment, so no confounder can influence assignment. Suppose one hundred patients get the drug and seventy recover, while one hundred get placebo and fifty five recover. The average treatment effect is seventy percent minus fifty five percent, fifteen percentage points.

• Coin flip assigns treatment
• Breaks the link to confounders
• ATE = difference in means

Behind this sits the counterfactual question: what would this patient's outcome have been without the drug? We can never observe both worlds for one person, which is the fundamental problem of causal inference. When experiments are impossible, adjustment methods can help, but only if we have measured the confounders. Never assume that a flexible model removes the problem.

• What would have happened otherwise?
• Observational data needs adjustment

Now fairness. A loan model approves applicants. Demographic parity asks that approval rates be equal across groups. In group A, sixty of one hundred are approved. In group B, thirty of one hundred. The rates are zero point six and zero point three, a gap of thirty points, so parity fails. But parity ignores whether the groups differ in genuine qualifications.

• Equal approval rates across groups
• Example: 60% versus 30%

Equalized odds conditions on the truth. Among people who truly would repay, the approval rate, the true positive rate, should match across groups, and so should the false positive rate. Say group A has fifty qualified applicants and forty are approved: a rate of zero point eight. Group B has forty qualified, and twenty are approved: zero point five. Qualified people in B are being missed.

• Equal true positive rate
• Equal false positive rate
• Example TPR 0.8 versus 0.5

You cannot usually satisfy every fairness definition at once. When the underlying base rates differ between groups, a classifier generally cannot be perfectly calibrated and also satisfy equalized odds, except in trivial cases. Fairness also trades against accuracy. So choose the metric by the harm: missed qualified applicants, or wrongly flagged ones. Document the choice, and measure it per group, as we will in lesson 65.

• Base rates differ
• Calibration and equalized odds conflict
• Pick based on harm and context

Last, robustness. An adversarial example is an input changed by a tiny, carefully chosen perturbation that makes the model fail, even though a human sees no difference. A panda image plus faint noise can be classified as a gibbon with high confidence. The model is highly sensitive in directions we never think about.

• Adversarial examples
• Perturbation invisible to humans

The fast gradient sign method is one line. Compute the gradient of the loss with respect to the input, not the weights. Take only its sign, plus one or minus one per pixel, multiply by a small epsilon, and add it to the input. This moves every pixel a tiny step in the direction that increases the loss the most.

• x_adv = x + ε · sign(∇ₓ L)
• Step along the loss gradient sign

Take three pixel values: zero point five, zero point two, zero point eight. Suppose the loss gradient has signs plus, minus, plus, and epsilon is zero point one. Each pixel moves by exactly zero point one in its sign direction. The result is zero point six, zero point one, zero point nine. Each pixel changed only slightly, yet across thousands of pixels the combined effect on the output is large.

• x = [0.50, 0.20, 0.80]
• Gradient signs [+, −, +]
• ε = 0.1 → [0.60, 0.10, 0.90]

The best-known defence is adversarial training: generate attacks during training and teach the model to resist them, usually at some cost in clean accuracy. Monitoring inputs and limiting access to the model also help. Be honest about limits, though: stronger attacks keep appearing, and robustness to one threat does not guarantee robustness to another.

• Adversarial training
• Input validation and monitoring
• No defence is complete

Let us recap. Separate aleatoric from epistemic uncertainty, and use conformal prediction for guaranteed coverage. Watch for covariate, label and concept shift, and flag anomalies. Explain models with permutation importance, partial dependence and LIME, but never mistake that for causation. Measure fairness per group, and test against adversarial attacks.

• Uncertainty and conformal
• Shift, anomalies, explanations
• Causality, fairness, robustness

Trustworthy models only matter if they are shipped and watched. In the next lesson, MLOps, we build reproducible pipelines, serve models, deploy them, and monitor for the very shifts we met today.

• Lesson 63: reproducible pipelines
• Serving, deployment, monitoring