ROC AUC by ranking
Hard20 min11 hidden tests
A lender's model gives each loan applicant a default risk score, and you know which applicants later defaulted. Write roc_auc(scores, labels) that returns the area under the ROC curve. Compute it from ranks: AUC is the chance that a randomly chosen defaulter gets a higher score than a randomly chosen applicant who repaid, where a tied pair counts as half.
Drag the scores
AUC = share of pairs where the defaulter scores higher| defaulted vs repaid | #1 (0.3) | #2 (0.6) | #4 (0.1) |
|---|---|---|---|
| #0 (0.8) | 1 | 1 | 1 |
| #3 (0.6) | 1 | ½ | 1 |
5.5 of 6 pairs are ordered right (a tie counts ½): AUC = 0.9167. From ranks: the defaulters’ ranks add up to 8.5, and (8.5 − 2 × 3 / 2) / (2 × 3) = 0.9167, the same.
AUC 0.9167.
Examples
One defaulter ties a repaid loan at 0.6
roc_auc([0.8, 0.3, 0.6, 0.6, 0.1], [1, 0, 0, 1, 0])
returns 0.916667
All scores tie, so the model knows nothing: 0.5
roc_auc([0.5, 0.5, 0.5], [1, 0, 0])
returns 0.5
Requirements
scoresis a list of n floats.labelsis a list of n ints, 1 for a default and 0 for a repaid loan. Both classes appear at least once.- Return the AUC as a float between 0.0 and 1.0.
- Rank the scores from lowest (rank 1) to highest. Tied scores all get the average of the ranks they span.
- AUC = (sum of the defaulters' ranks - n_pos * (n_pos + 1) / 2) / (n_pos * n_neg), where n_pos and n_neg count the defaulters and the repaid loans.
The idea behind it
The ROC curve plots the true positive rate against the false positive rate as a decision threshold slides from high to low. The area under it, AUC, has a simpler meaning: pick one positive and one negative at random, and AUC is the chance the positive gets the higher score. A tie counts as half. So 1.0 means every defaulter outranks every repaid loan, 0.5 means the scores carry no signal, and 0.0 means the order is exactly backwards. AUC ignores the threshold and the scale of the scores. Only their order matters. Checking every pair costs n_pos * n_neg comparisons, which is too slow for millions of loans. The Mann-Whitney U statistic gives the same number from one sort: rank the scores, add up the positives' ranks, and subtract the smallest sum they could have. Credit, fraud and medical teams report AUC because it compares models before anyone picks a threshold.