Step 1: Count the four outcomes
Brightline's website runs every order through a fraud model that gives it a score from 0 to 1.
Today the shop holds orders scoring 0.5 or more for a person to check, because 0.5 is the default.
Nobody chose it. Fraud losses keep rising, and the manager wants to know whether the model is bad or
the setting is. orders_valid.csv has last week's 20,000 orders, each with its score and whether it
turned out to be fraud. costs.md says what each kind of mistake costs.
Any threshold splits the orders four ways: fraud held (true positive), genuine held (false positive), fraud missed (false negative) and genuine shipped (true negative). Every metric is built from those four counts.
1. Write confusion(y, scores, threshold). held = scores >= threshold gives one True/False per
order. Combine it with the outcome: (held & (y == 1)).sum() counts true positives, and ~held means
"not held".
2. Write rates(c): accuracy, precision (of the held orders, the share that are fraud),
recall (of the fraud, the share held) and flag rate (the share of all orders held).
3. Answer the question below, then Run.
fraud.py, the file you edit61 lines
"""A fraud screen: pick the threshold from what mistakes cost, not from 0.5."""
import numpy as np
import pandas as pd
from sklearn.metrics import average_precision_score, precision_recall_curve, roc_auc_score, roc_curve
COST_MISSED_FRAUD = 75 # £, a false negative (costs.md)
COST_FALSE_ALARM = 6 # £, a false positive
MAX_FLAG_RATE = 0.05 # the review team checks at most 5% of orders
THRESHOLDS = np.round(np.arange(0.01, 1.0, 0.01), 2) # 0.01, 0.02, ... 0.99
def load(path):
df = pd.read_csv(path)
return df["is_fraud"].to_numpy(), df["score"].to_numpy()
# ---------- Step 1: count the four outcomes ----------
def confusion(y, scores, threshold):
"""{"tp", "fp", "fn", "tn"} as ints when orders with score >= threshold are held."""
# TODO (Step 1): held = scores >= threshold (a boolean array, one per order)
# tp: held and fraud (y == 1) fp: held and genuine (y == 0)
# fn: not held (~held) and fraud tn: not held and genuine
# Count each with (condition).sum() and return plain ints: int(...)
raise NotImplementedError("Step 1: write confusion()")
def rates(c):
"""{"accuracy", "precision", "recall", "flag_rate"} from a confusion dict, each rounded to 3.
precision is 0.0 when nothing is held."""
# TODO (Step 1): from c = {"tp", "fp", "fn", "tn"}:
# accuracy = (tp + tn) / all orders
# precision = tp / (tp + fp) (0.0 when nothing is held)
# recall = tp / (tp + fn)
# flag_rate = (tp + fp) / all orders
raise NotImplementedError("Step 1: write rates()")
# ---------- Step 2: two curves, two summaries ----------
def curves(y, scores, path="curves.png"):
"""Plot the ROC curve and the precision-recall curve side by side into path, and return
{"roc_auc", "average_precision"}, each rounded to 3."""
raise NotImplementedError("curves() arrives in Step 2")
# ---------- Step 3: what a threshold costs ----------
def cost(y, scores, threshold):
"""Total £ cost of the mistakes at this threshold, as an int."""
raise NotImplementedError("cost() arrives in Step 3")
def best_threshold(y, scores, max_flag_rate=None):
"""The threshold in THRESHOLDS with the lowest cost. With max_flag_rate, only thresholds that hold
at most that share of orders count. On a tie, the lower threshold wins."""
raise NotImplementedError("best_threshold() arrives in Step 3")
# ---------- Step 5: next week ----------
def weekly_report(valid_path="orders_valid.csv", test_path="orders_test.csv"):
"""Choose the threshold on last week's orders (with the review limit), then measure it on this week's.
{"threshold", "cost", "cost_at_half", "cost_no_screen", "recall", "flag_rate"} for THIS week."""
raise NotImplementedError("weekly_report() arrives in Step 5")costs.mdorders_valid.csvtry_it.py