A model with 99 percent accuracy that catches zero fraud? Here is how accuracy fools you on imbalanced data, and how a confusion matrix exposes it.
Introduction
99% Accurate, Zero Caught
Your model is ninety-nine percent accurate. And it catches zero fraud.
A Rare Event Dataset
Imagine a fraud dataset with ten thousand transactions. Only one percent are fraud. That is just one hundred cases.
- 10,000 transactions
- 100 frauds
- 1% positive
The Laziest Model
Now build the laziest model possible. It ignores every input and always predicts not fraud.
- Ignores all inputs
- Always predicts negative
Counting Right and Wrong
It is right on all nine thousand nine hundred legitimate cases. Wrong only on the one hundred frauds.
- 9,900 correct
- 100 missed
The Accuracy Formula
Accuracy is correct predictions divided by total predictions. Nine thousand nine hundred over ten thousand is ninety-nine percent.
- Correct divided by total
- 9,900 / 10,000 = 0.99
The Confusion Matrix
Looks great. So open the confusion matrix. Here is the truth.
- True positives: 0
- False negatives: 100
Recall Is Zero
Recall is true positives divided by all actual frauds. Zero over one hundred is zero. Not one fraud caught.
- Recall = TP / (TP + FN)
- 0 / 100 = 0
Do Not Trust Accuracy Alone
With imbalanced data, accuracy hides failure. Always check the confusion matrix, precision, and recall. Follow for the next lesson.
- Check the confusion matrix
- Use precision, recall, F1