Precision and recall at review capacity
Medium12 min10 hidden tests
A fraud model scores every card transaction, and the review team can check only the k highest-scoring ones each day. Write precision_recall_at_k(scores, labels, k) that flags the top k scores and returns the precision and recall of those flags. When two scores tie, the earlier transaction ranks higher.
Set the review capacity
flag the k highest scores, ties to the earlier oneThe team reviews transactions 0 and 2: 1 of 2 is fraud, precision 0.5. They catch 1 of the 3 frauds, recall 0.333.
Precision 0.5, recall 0.333.
Examples
The team reviews the two highest scores
precision_recall_at_k([0.9, 0.2, 0.75, 0.4, 0.6], [1, 0, 0, 1, 1], 2)
returns (0.5, 0.333333)
A tie at 0.5 goes to the earlier transaction
precision_recall_at_k([0.8, 0.5, 0.5, 0.1], [1, 0, 1, 0], 2)
returns (0.5, 0.5)
Requirements
scoresis a list of floats,labelsa list of 0 or 1 of the same length (1 means fraud), andkis between 1 and the number of transactions.- Return a tuple
(precision, recall)of two floats. - Precision is the share of flagged transactions that are fraud. Recall is the share of all fraud that got flagged, and it is 0.0 when there is no fraud.
- Ties in score go to the lower index.
The idea behind it
A score threshold is often set by how much a team can review. A fraud team, a content moderation queue or a sales team can only review so many cases a day, so the model's job is to put the right cases at the top. Two numbers describe that top list. Precision at k is the share of the k reviewed cases that are real: how much of the team's time goes to true fraud. Recall at k is the share of all real cases that made the list: how much fraud gets caught at all. Raising k usually lifts recall and lowers precision. Teams pick k from their capacity, then compare models by these two numbers at that k.