Credit Risk, Fairness and Anomaly Detection: Capstone 3 and 4

A bank model can be ninety-five percent accurate and still be a bad model. It can approve the wrong people, reject the right ones, and treat two groups very differently. Today we build two capstone projects where accuracy alone is never the goal.

• Credit risk
• Fairness audit
• Anomaly detection

In the last lesson we built house-price prediction and churn with a full project template. Here we do not repeat that template. We focus on what is new: costs, imbalance, calibration, fairness, and detecting rare events without labels.

• Template from lesson 64
• New: costs and fairness
• New: no labels

Start with the decision, not the model. A lender must decide whether to approve a loan. If it approves someone who defaults, it loses the money lent. If it rejects someone who would have repaid, it loses the interest it would have earned.

• Approve or reject
• Two kinds of mistakes
• Each mistake has a price

Put numbers on it. Suppose a default costs eight thousand dollars, while a missed good customer costs six hundred dollars of lost profit. A false approval is more than thirteen times as costly as a false rejection. This ratio will drive our threshold later.

• Default: 8000 dollars
• Missed customer: 600 dollars
• Ratio about 13 to 1

Also decide what data you may use. A public credit dataset gives us income, debt, history of late payments, loan amount and so on. Features must be known at application time. Anything recorded after the loan, such as collection notes, is leakage, as we saw in lesson 35.

• Features at application time
• No post-loan information
• Leakage check first

Defaults are rare, perhaps seven percent of loans. A model that predicts everyone repays gets ninety-three percent accuracy and catches zero defaults. That is why accuracy misleads, and why we track precision, recall and PR-AUC instead.

• 7 percent defaults
• Always-repay scores 93 percent
• Recall is zero

There are three common fixes. Use class weights so a default counts more in the loss. Resample the training data, by oversampling defaults or undersampling repayers. Or leave the data alone and fix the threshold afterward. Resample only inside the training fold, never before splitting.

• Class weights
• Resample training fold only
• Or adjust threshold

Our baseline is logistic regression with scaled features, from lessons 15 and 41. It is simple, fast and easy to explain, which matters in lending. Say it reaches a PR-AUC of point three five on a held-out test set where random guessing would give about point zero seven. That is our bar to beat.

• Scaled features
• Weighted loss
• PR-AUC 0.35 versus 0.07

Next a gradient boosted model, as in lesson 42. Trees capture interactions like high debt combined with recent late payments. Suppose it reaches point four two. It is better, but it is harder to explain, and we must check it did not just memorize, using cross validation and a fresh test set touched once.

• Captures interactions
• Example PR-AUC 0.42
• Test set used once

A score is not yet a probability. Boosted models often output scores that say eighty percent when only fifty-five percent of such applicants really default. Lesson 16 showed calibration: group predictions into bins and compare predicted rate to observed rate.

• Score is not probability
• Bin and compare
• Gap means overconfident

We fix it by fitting a calibrator on a separate validation set, using Platt scaling or isotonic regression. After that the curve hugs the diagonal. This matters because the cost calculation needs real probabilities, not arbitrary scores.

• Fit on validation data
• Curve hugs diagonal
• Needed for cost math

Now the threshold, derived from costs. Approve when expected loss from approving is less than expected loss from rejecting. Approving costs p times eight thousand. Rejecting costs one minus p times six hundred. They are equal when p equals six hundred over eight thousand six hundred, about point zero seven.

• Approve cost: 8000 p
• Reject cost: 600 (1 minus p)
• Break-even p about 0.07

So we reject anyone whose calibrated default probability is above about seven percent. A common mistake is to use point five by habit. At point five, we would approve almost every defaulter, and the cost table would show a large loss. Always compare thresholds by total expected cost on validation data.

• Cost curve over thresholds
• Minimum near 0.07
• 0.5 is a habit, not a rule

Now the audit. Suppose applicants belong to two groups, A and B, recorded only for auditing. A model can be accurate overall and still behave differently across groups. We compute metrics separately for each group, never just on the whole population.

• Groups for auditing
• Metrics per group
• Not just overall

Demographic parity asks whether approval rates are equal. Say group A is approved at seventy percent and group B at fifty-six percent. The ratio is point eight, which is exactly the four-fifths rule of thumb used as a screening signal in some contexts. It is a flag to investigate, not a verdict.

• Equal approval rates?
• 70 percent versus 56 percent
• Ratio 0.8, a flag only

Demographic parity ignores who actually repays. If base default rates differ, equal approval could force unequal error rates. Equalized odds instead asks for equal true positive and false positive rates across groups, so that among people with the same real outcome, the model treats groups alike.

• Equal true positive rate
• Equal false positive rate
• Conditioned on real outcome

Concrete numbers help. Among people who really repay, group A is wrongly rejected twelve percent of the time and group B twenty-two percent. That ten point gap is a harm: creditworthy people denied. Reporting false rejection and false approval rates per group makes such gaps visible.

• False rejection: 12 versus 22
• Creditworthy people denied
• Report both error types

Removing the protected attribute does not make the model fair. Features like postal code, shopping patterns or type of phone can act as proxies, because they correlate with group membership. A test is to try predicting the group from the other features. If that works well, proxies are present.

• Dropping the attribute is not enough
• Postal code can proxy group
• Test: predict group from features

As lesson 62 explained, these criteria generally cannot all hold at once when base rates differ. Calibration within groups, equal error rates and equal approval rates pull in different directions. So fairness is a choice about which harm to prioritize, made with stakeholders, not a number the model optimizes for free.

• Criteria conflict
• Base rates differ
• A choice, not a free lunch

Mitigation options exist. Before training, reweight or resample data. During training, add a fairness penalty. After training, use separate thresholds per group to equalize error rates. Each one costs some accuracy or profit, so measure the trade-off explicitly instead of guessing.

• Before, during, after training
• Each has a cost
• Measure the trade-off

But mitigation has limits. It cannot fix biased labels, since past lending decisions shaped who defaulted. It does not guarantee fairness on new populations. And the law matters: in many places, lending is regulated, using protected attributes directly may be illegal, and applicants may have a right to an explanation. Check with legal experts for your jurisdiction.

• Biased labels remain
• Populations shift
• Regulation applies

Finish the project with a model card. It records the intended use, the training data, the metrics overall and per group, the chosen threshold and why, known limitations, and a monitoring plan. Anyone who inherits the model should know what it is for and where it should not be used.

• Use and data
• Metrics per group
• Limits and monitoring

Now project four: anomaly detection. Think of credit card transactions, where fraud is a tiny fraction and labels are scarce or late. We train mostly on normal behavior and flag points that look unlike it. It is unlabeled at training time, but we still need labels to evaluate.

• Fraud or faults are rare
• Few or late labels
• Learn normal, flag different

We use a public credit card fraud dataset, where fewer than one percent of transactions are fraudulent. Features are anonymized components plus time and amount. Split by time if you can, because in real life the model predicts the future. And fit scalers only on training data.

• Under 1 percent fraud
• Split by time
• Fit scalers on train only

First, the isolation forest. It builds random trees that split features at random values. Odd points sit alone and get isolated after few splits, while normal points need many. The anomaly score is based on average path length: short path means suspicious.

• Random splits
• Short path means odd
• Score from average depth

Second, Gaussian scoring with the Mahalanobis distance from lesson 38. We fit a mean and covariance to normal data. The distance measures how many standard deviations away a point is, accounting for correlation. A point can look ordinary in each feature alone, but be strange in combination.

• Fit mean and covariance
• Accounts for correlation
• Odd combinations stand out

A tiny example. Two correlated features, with mean zero, variance one, and correlation point nine. The point one and minus one has Euclidean length about one point four, which seems normal. But the Mahalanobis squared distance is two divided by one point nine, wait, it is two times one point nine over point nineteen, about twenty. That is wildly unusual.

• Correlation 0.9
• Point (1, minus 1)
• Squared distance about 20

Third, an autoencoder from the deep learning lessons. It compresses each transaction to a small code and reconstructs it. Train it on normal data only. Normal points rebuild with low error, while anomalies, which the network never learned to represent, rebuild badly. Reconstruction error is the score.

• Compress then rebuild
• Train on normal data
• Error is the score

A warning: if the autoencoder is too big, it learns to copy everything, anomalies included. Keep the bottleneck small, use regularization, and watch validation reconstruction error. Also remember that Mahalanobis assumes roughly Gaussian data, so heavy tails or several clusters can break it.

• Too big copies everything
• Gaussian assumption can fail
• Check on validation

Evaluate with ranking metrics. ROC-AUC looks flattering on rare events because true negatives are plentiful. PR-AUC is stricter. And precision at k asks: among the top k highest scores, how many are real anomalies? If analysts can only review one hundred alerts a day, k equals one hundred.

• ROC-AUC flatters rare events
• PR-AUC is stricter
• Precision at review capacity k

Suppose precision at one hundred is point three for the isolation forest, point four five for Mahalanobis and point five two for the autoencoder. Differences like this must be checked across several random seeds and time windows, because with so few positives the numbers are noisy. Do not crown a winner from one run.

• Example precision at 100
• Repeat over seeds
• Noise from few positives

Alert thresholds are cost design. Each false alarm costs analyst time, say five dollars. Each missed fraud costs two hundred. If a threshold gives a hundred alerts with thirty frauds, cost is seventy false alarms times five, plus whatever frauds were missed below the line. Sweep the threshold and pick the minimum total cost.

• False alarm: 5 dollars
• Missed fraud: 200 dollars
• Pick minimum total cost

Alert fatigue is real. If most alerts are false, people stop trusting them. Use tiers: auto-block the highest scores, send mid scores to review, and only log the rest. Then monitor the alert rate and score drift over time, as in lesson 63, because anomalies and normal behavior both change.

• Block, review, log
• Prevent alert fatigue
• Monitor drift

Let us recap. For credit risk, define the decision and its costs, handle imbalance, compare a logistic baseline with boosting, calibrate, and set the threshold from costs. Audit fairness by group, remember proxies, trade-offs, legal limits, and write a model card.

• Costs drive threshold
• Audit every group
• Document with a model card

For anomalies, learn normal behavior without labels using isolation forest, Mahalanobis distance or an autoencoder. Evaluate with precision at k and PR-AUC, and design alerts around false-alarm cost. Next lesson, we turn to text with spam classification and NLP text classification.

• Three detectors
• Precision at k, PR-AUC
• Next: spam and text