Preporato

Confusion counts for a spam filter

Easy8 min10 hidden tests

A support inbox runs a spam filter. For a day of messages you have the true labels (1 for spam, 0 for a real message) and the filter's predictions in the same order. Write confusion_counts(labels, preds) that returns the four counts of the confusion matrix: true positives, false positives, false negatives and true negatives.

Move the threshold

flagged when score ≥ threshold
FraudNot fraud
FlaggedNot flaggedFraud
7true positives
3false negatives: missed
Not fraud
4false positives: false alarms
10true negatives

Precision 0.64: of the 11 flagged, 7 are fraud. Recall 0.70: 7 of the 10 fraud cases get caught.

Threshold 0.50. 7 true positives, 4 false positives, 3 false negatives, 10 true negatives.

Examples

Two spam caught, one missed, one real message flagged

confusion_counts([1, 0, 1, 1, 0, 0], [1, 0, 0, 1, 1, 0])

returns {'tp': 2, 'fp': 1, 'fn': 1, 'tn': 2}

The filter flags two real messages

confusion_counts([0, 0, 0, 1], [1, 1, 0, 1])

returns {'tp': 1, 'fp': 2, 'fn': 0, 'tn': 1}

Requirements

  • labels and preds are lists or 1-D arrays of 0s and 1s with the same length. They may be empty.
  • Return a dict with the keys "tp", "fp", "fn" and "tn", each an int.
  • A positive is a prediction of 1. It is true when the label is also 1. A negative is a prediction of 0, and it is false when the label is 1.
  • For empty inputs every count is 0.
The idea behind it

A classifier that says yes or no can be right or wrong in two ways each. The confusion matrix, a 2 x 2 table, counts all four outcomes. A true positive (TP) is a yes that was right: spam flagged as spam. A false positive (FP) is a yes that was wrong: a real message sent to the spam folder. A false negative (FN) is a no that was wrong: spam that reached the inbox. A true negative (TN) is a no that was right. The four counts add up to the number of messages. Each type of error costs something different. A false positive can hide a customer's refund request, while a false negative only annoys an agent. That is why teams read the four cells before they read any single score. Precision, recall, F1 and accuracy are all ratios of these counts.

pythonCtrl+Enter runs
import numpy as np

def confusion_counts(labels, preds):
"""Count the four cells of the confusion matrix.

labels: list or 1-D array of n ints, 1 for spam and 0 for a real message
preds: list or 1-D array of n ints, the filter's call for each message
Returns a dict {"tp": int, "fp": int, "fn": int, "tn": int}.
"""
raise NotImplementedError

Stuck? Reveal one rung at a time

Worked solution

Give it a real try first. Then read the solution and why each line is there.