Classification Metrics, Thresholds and Calibration Explained Visually

Imagine a model that screens patients for a rare disease and scores ninety eight percent accuracy. Sounds fantastic. Now imagine it has never found a single sick patient. That is not a hypothetical trick. It is what happens when we judge a classifier with the wrong number.

• Accuracy can lie
• Imbalance hides failure

In the last episode we built logistic regression and softmax, models that output a probability. But a probability is not a decision, and a decision is not automatically a good one. In this episode we learn how to measure a classifier honestly, how to choose a threshold, and how to check whether its probabilities can be trusted.

• Last time: logistic regression
• Output is a probability
• Now: judge it properly

Here is the plan. We start with the confusion matrix, the table behind every metric. Then we see why accuracy fails, and meet precision, recall and F1. We move on to thresholds, ROC and precision recall curves, cost based decisions, imbalance, and calibration. We finish with scikit-learn code and a rule for matching metric to task.

• Confusion matrix
• Precision, recall, F1
• ROC, PR, calibration

A binary classifier says yes or no, and reality is also yes or no. That gives exactly four outcomes. A true positive means we said yes and it was yes. A true negative means we said no and it was no. A false positive is a false alarm. A false negative is a miss.

• Prediction: yes or no
• Truth: yes or no
• Four combinations

A trick for the names: the second word is what the model predicted, and the first word says whether that prediction was right. So false positive means the model predicted positive and was wrong. False negative means it predicted negative and was wrong. Keep this decoder in mind, because every metric is just a ratio of these four counts.

• Second word: what we predicted
• First word: were we right?

Let us use one example all episode. A thousand patients, of whom twenty are truly sick. Our model flags thirty patients. Of those, twelve are really sick, so twelve true positives, and eighteen are healthy, so eighteen false positives. It missed eight sick patients, the false negatives, and correctly cleared nine hundred sixty two healthy people.

• 1000 patients
• 20 are sick
• TP 12, FN 8, FP 18, TN 962

Always check that the cells add up. Twelve plus eight is twenty actual positives. Eighteen plus nine hundred sixty two is nine hundred eighty actual negatives. Together a thousand. The predicted positive column adds to thirty. This takes five seconds and catches a surprising number of bugs, like mixing up which axis is truth and which is prediction.

• Rows sum to actual counts
• All cells sum to n
• 12 + 8 + 18 + 962 = 1000

Accuracy is the fraction of predictions that were correct. Add the true positives and true negatives, divide by everything. Here twelve plus nine hundred sixty two is nine hundred seventy four, over a thousand, so ninety seven point four percent. It looks great. Now watch what happens when we compare it with a model that does nothing.

• Correct / total
• (TP + TN) / n
• 974 / 1000 = 97.4%

Consider a lazy model that always predicts healthy. It gets all nine hundred eighty healthy patients right and all twenty sick patients wrong. Accuracy: nine hundred eighty over a thousand, ninety eight percent. The useless model beats our real one. Accuracy is dominated by the majority class, so rare events vanish inside it.

• Always predict healthy
• TP 0, FN 20, FP 0, TN 980
• Accuracy 98%

This is why we always compute the majority class baseline. If ninety eight percent of cases are negative, ninety eight percent accuracy is the floor, not an achievement. Fraud, disease, machine failures and spam bursts are all rare in practice, and rare is exactly where we care most. We need metrics that look at the positive class directly.

• Majority-class accuracy is a floor
• Fraud, disease, defects are rare
• Always compare to baseline

Recall asks: of all the truly sick patients, how many did we catch? That is true positives over true positives plus false negatives. Twelve over twenty is zero point six. We found sixty percent of the sick. Recall is also called sensitivity or true positive rate. The lazy model has recall zero, which exposes it immediately.

• TP / (TP + FN)
• 12 / 20 = 0.60
• Also: sensitivity, TPR

Precision asks a different question: when the model raises an alarm, how often is it right? True positives over true positives plus false positives. Twelve over thirty is zero point four. Only forty percent of flagged patients are actually sick. Precision is about the cost of false alarms, recall is about the cost of misses.

• TP / (TP + FP)
• 12 / 30 = 0.40
• Trust in a positive flag

People constantly swap these two. Here is the cleanest way to remember. Recall divides by the actual positives, the top row of the matrix. Precision divides by the predicted positives, the left column. Same numerator, true positives, but a different denominator. Just ask yourself: am I conditioning on reality, or on what the model said?

• Recall denominator: actual positives
• Precision denominator: predicted positives

There is a mirror image for the negative class. Specificity is true negatives over all actual negatives: nine hundred sixty two over nine hundred eighty, about zero point nine eight. The false positive rate is one minus that, eighteen over nine hundred eighty, about zero point zero one eight. These two will matter again when we draw the ROC curve.

• Specificity = TN / (TN + FP)
• FPR = FP / (FP + TN)
• 962 / 980 = 0.98

Sometimes you want one number combining both. F1 is the harmonic mean of precision and recall: two times precision times recall, over precision plus recall. With zero point four and zero point six, the numerator is zero point four eight and the denominator is one, so F1 is zero point four eight. The arithmetic mean would say zero point five.

• Harmonic mean
• F1 = 2PR / (P + R)
• F1 = 0.48

Why not a plain average? Because a plain average hides disasters. A model with precision one and recall zero point zero one would average about zero point five, yet it finds almost nothing. Its F1 is about zero point zero two. The harmonic mean is dragged down by the weaker number, so you cannot cheat by maximising only one.

• Punishes imbalance between P and R
• P = 1, R = 0.01 gives F1 about 0.02

Logistic regression outputs a probability. To get a label we choose a threshold: predict positive if the probability is at least t. The default of zero point five is just a convention. It makes sense only when both mistakes cost the same and the classes are balanced. Moving the threshold changes every metric we just computed.

• Model outputs p
• Rule: predict 1 if p ≥ t
• Default t = 0.5 is arbitrary

Let us shrink the problem so we can compute everything by hand. Eight samples sorted by score. Positives have scores zero point nine five, zero point eight, zero point five five and zero point three five. Negatives have scores zero point six five, zero point four, zero point two and zero point one. Four positives, four negatives.

• 8 samples, 4 positive
• Scores: 0.95 0.80 0.65 0.55 0.40 0.35 0.20 0.10
• Sorted by score

At threshold zero point seven, we flag two samples, both positive. Precision is one, recall is one half. At zero point five we flag four: three true, one false. Precision and recall are both three quarters. At zero point three we flag six: all four positives plus two negatives. Recall is one, precision falls to two thirds. Lower threshold, more recall, less precision.

• t = 0.7: TP 2, FP 0
• t = 0.5: TP 3, FP 1
• t = 0.3: TP 4, FP 2

Instead of picking one threshold, plot them all. The ROC curve has false positive rate on the x axis and true positive rate, which is recall, on the y axis. Each threshold gives one point. At zero point seven: zero and zero point five. At zero point five: zero point two five and zero point seven five. At zero point three: zero point five and one.

• x: false positive rate
• y: true positive rate
• One point per threshold

A perfect model reaches the top left corner: all positives caught, no false alarms. Random guessing follows the diagonal. The further a curve bows toward the top left, the better the model ranks positives above negatives. Each point along the curve is a different operating choice, so the curve is a menu of trade-offs.

• Top-left corner is perfect
• Diagonal is random guessing
• Curve shows trade-off

AUC is the area under the ROC curve. One is perfect, zero point five is random. Notice that it does not depend on any single threshold. For our toy data the area comes to thirteen sixteenths, which is zero point eight one two five. But there is a more intuitive way to read that number, and it comes from counting pairs.

• Area under ROC curve
• 1.0 perfect, 0.5 random
• Our toy model: 0.8125

AUC equals the probability that a random positive scores higher than a random negative. Check it: sixteen pairs. The positive at zero point nine five beats all four negatives. Zero point eight also beats all four. Zero point five five beats three. Zero point three five beats two. That is thirteen correct pairs out of sixteen, so AUC is zero point eight one two five. Same answer.

• Pick one positive, one negative
• AUC = P(score pos > score neg)
• 13 of 16 pairs correct

AUC only measures ranking. If you multiply every score by a constant that keeps the order, AUC does not change, even though the probabilities are now nonsense. It also says nothing about which threshold to use. And with very rare positives, the false positive rate can look tiny even when thousands of false alarms swamp the true hits.

• Measures ranking only
• Ignores calibration
• Can look rosy under imbalance

The precision recall curve plots precision against recall as the threshold sweeps. Our toy points: recall one half with precision one, recall three quarters with precision three quarters, recall one with precision two thirds. The random baseline is a flat line at the fraction of positives, here one half. A perfect model sits in the top right.

• x: recall
• y: precision
• Baseline = positive rate

Back to our patients. Eighteen false positives out of nine hundred eighty negatives is a false positive rate under two percent, which makes the ROC curve look healthy. But precision is forty percent, meaning most alarms are false. Precision ignores true negatives, so the huge pile of easy negatives cannot hide the problem. That is why PR curves are preferred for rare positive classes.

• ROC uses true negatives
• PR ignores true negatives
• FPR hides false alarms

Average precision summarises the PR curve in one number, roughly a weighted mean of precision across recall levels. Read it against the baseline, which equals the positive rate. If only two percent of cases are positive, a score of zero point three is a fifteen times improvement over random, even though the number looks small. Context decides what is good.

• Summary of the PR curve
• Compare against positive rate
• Not the same as AUC of ROC

Real decisions have costs. Suppose a missed sick patient costs five hundred units, and a false alarm leading to an extra test costs twenty. Then total cost is five hundred times false negatives plus twenty times false positives. Our current model has eight misses and eighteen false alarms: four thousand plus three hundred sixty, so four thousand three hundred sixty.

• Missed disease: cost 500
• False alarm: cost 20
• Costs set the threshold

Suppose we lower the threshold. Now we catch seventeen of twenty sick patients, but raise ninety false alarms. Cost: three misses times five hundred is fifteen hundred, plus ninety times twenty is eighteen hundred, total three thousand three hundred. Better. Lower it too far, say one miss and three hundred false alarms, and the cost jumps to six thousand five hundred. There is a sweet spot.

• t lower: TP 17, FN 3, FP 90
• Cost = 1500 + 1800 = 3300
• Too low: FP 300, FN 1 gives 6500

There is a neat result. Predict positive when the expected cost of a miss exceeds the expected cost of a false alarm. That gives p times five hundred greater than one minus p times twenty, so p greater than twenty over five hundred twenty, about zero point zero three eight. A tiny threshold, justified by cost. But this only works if the probabilities are calibrated, which we will get to.

• Predict 1 if p > C_FP / (C_FP + C_FN)
• 20 / 520 ≈ 0.038
• Requires calibrated p

One way to handle imbalance is to change the loss. Class weights make each rare example count more. The standard balanced formula is n over k times the class count, with k classes. Positives get one thousand over two times twenty, which is twenty five. Negatives get about zero point five one. Now both classes contribute equally to the training loss.

• weight = n / (k × n_c)
• Positive: 1000 / (2×20) = 25
• Negative: 1000 / (2×980) ≈ 0.51

Another route is resampling. Oversampling duplicates minority examples. Undersampling throws away majority examples, which is fast but loses information. SMOTE creates synthetic minority points by interpolating between a real minority sample and one of its nearest minority neighbours. All of these rebalance what the model sees in training.

• Oversample the minority
• Undersample the majority
• SMOTE: synthetic neighbors

A critical warning. Resample only the training data. The validation and test sets must keep the real class ratio, because that is what the model will face in production. If you oversample before splitting, duplicated points land on both sides, the test score is inflated, and you have leaked data. Do it inside each cross validation fold. Also remember weights and resampling shift probabilities, which breaks calibration.

• Never touch validation or test
• Do it inside cross-validation folds
• Leakage inflates scores

A model is calibrated if its probabilities mean what they say. Among all cases where it predicts zero point eight, about eighty percent should turn out positive. Think of a weather forecaster who says seventy percent chance of rain: over many such days, it should rain on about seventy percent. This is a separate property from ranking or accuracy.

• Of cases scored 0.8, about 80% positive
• Probability as frequency
• Different from accuracy

To check, we draw a calibration curve. Sort predictions into bins. In each bin, compute the average predicted probability and the actual fraction of positives. Say the zero point seven to zero point eight bin has two hundred samples and one hundred forty are positive: observed seventy percent against a predicted seventy five. Plot every bin. A perfect model lies on the diagonal.

• Bin predictions
• x: mean predicted p
• y: observed fraction positive

If the curve falls below the diagonal, the model is overconfident: it says ninety percent but only seventy percent are positive. Above the diagonal means underconfident. Logistic regression is often reasonably calibrated when it fits well, while many other models, such as boosted trees or margin based ones, commonly need correction. It is worth always checking rather than assuming.

• Below diagonal: overconfident
• Above diagonal: underconfident
• Many tree and boosted models drift

The Brier score condenses probability quality into one number: the mean squared difference between predicted probability and the true label, zero or one. Take four predictions. Point nine for a positive gives zero point zero one. Point eight for a negative gives zero point six four. Point three for a negative gives zero point zero nine. Point six for a positive gives zero point one six. The sum is point nine, divided by four, so zero point two two five.

• Mean of (p − y)²
• Lower is better
• Example: 0.225

If calibration is poor, we can fix it after training. Platt scaling fits a small sigmoid that maps raw scores to probabilities. Isotonic regression fits a flexible, non decreasing step function, which needs more data. Either way, fit the calibrator on data the model was not trained on, otherwise you will just calibrate to the training set. Ranking, and so AUC, mostly stays the same.

• Platt scaling: sigmoid on scores
• Isotonic regression: monotone map
• Fit on held-out data

Now scikit-learn. The confusion matrix function takes true labels and predicted labels, and returns the table. Note that rows are actual and columns are predicted, with the negative class first. Classification report prints precision, recall and F1 per class. Run it on our example and you will see zero point four, zero point six and zero point four eight, matching our hand calculations.

• confusion_matrix
• classification_report
• precision, recall, f1_score

For curves, pass the true labels and the predicted probabilities, not the hard labels. ROC curve returns false positive rates, true positive rates and thresholds. ROC AUC score on our toy data returns zero point eight one two five. Precision recall curve and average precision score do the same for the PR view. A very common bug is passing predicted labels, which collapses the curve to one point.

• roc_curve, roc_auc_score
• precision_recall_curve
• average_precision_score

Logistic regression accepts class weight set to balanced, which applies the formula we derived. Calibration curve returns observed fractions and mean predictions per bin, ready to plot. Brier score loss gives the Brier number. Calibrated classifier CV wraps any model and calibrates with cross validation, using method sigmoid or isotonic. To apply a custom threshold, compare the probabilities to your chosen cutoff yourself.

• class_weight='balanced'
• calibration_curve, brier_score_loss
• CalibratedClassifierCV

Here is the practical rule. If misses are dangerous, as in screening, prioritise recall. If false alarms are expensive, as in a spam folder deleting real mail, prioritise precision. If positives are rare, use PR curves and average precision. If you will use the probabilities directly, for pricing or risk, check calibration and Brier score. Balanced and symmetric costs are the only case where accuracy is safe.

• Misses costly: recall
• False alarms costly: precision
• Rare positives: PR, AP
• Probabilities used: calibration, Brier

Let us collect the classic mistakes. Reporting only accuracy on imbalanced data. Choosing the threshold on the test set, which leaks. Resampling before splitting. Feeding hard labels into an AUC function. And trusting probabilities from a model that was never calibrated. Pick the threshold on validation data, report on untouched test data, and always show the confusion matrix.

• Reporting only accuracy
• Tuning threshold on test set
• Resampling before splitting
• Ignoring calibration

Recap. Everything starts from the confusion matrix. Accuracy fails under imbalance, so we use precision, recall and F1. Thresholds trade one against the other, ROC and PR curves show every trade-off, and AUC is the chance a positive outranks a negative. Costs choose the threshold, weights and resampling handle imbalance, and calibration tells us whether probabilities can be trusted.

• Confusion matrix is the foundation
• P, R, F1; ROC, PR, AUC
• Costs choose thresholds
• Calibrate before trusting p

In the next episode we tackle regularization and feature engineering: how to stop models from overfitting, what ridge and lasso really do geometrically, and how to build better inputs. Thanks for watching.

• Next: Regularization and Feature Engineering
• Subscribe for the full series