Preporato

Expected calibration error

Medium14 min11 hidden tests

A delivery app predicts the chance that each order arrives late, and the support team plans its shifts from those numbers. If the model says 0.8, about 80 percent of those orders should be late. Write expected_calibration_error(probs, labels, n_bins) that measures how far the predicted chances are from what happened, using equal-width bins.

Change the number of bins

ECE = Σ (count / n) × |late share − mean predicted|
Late shareMean predictedGapPerfect calibration

20 orders in 5 equal bins (the small numbers are the counts). ECE = 3/20 × 0.09 + 4/20 × 0.03 + 3/20 × 0.18 + 4/20 × 0.17 + 6/20 × 0.06 = 0.098.

Expected calibration error 0.098.

Examples

Two bins, each a little off

expected_calibration_error([0.1, 0.2, 0.8, 0.9, 0.7], [0, 1, 1, 1, 0], 2)

returns 0.22

A calibrated model scores 0.0

expected_calibration_error([0.75, 0.75, 0.75, 0.75, 0.25, 0.25, 0.25, 0.25], [1, 1, 1, 0, 0, 0, 0, 1], 2)

returns 0

Requirements

  • probs is a list of n floats in [0, 1], labels a list of n ints (1 if the order was late), and n_bins an int of 1 or more. n is at least 1.
  • Bin b covers [b / n_bins, (b + 1) / n_bins). Put a probability p in bin min(floor(p * n_bins), n_bins - 1), so p = 1.0 lands in the last bin.
  • For each bin that holds orders, take the absolute gap between the share of late orders and the mean predicted probability. Weight it by the bin's count divided by n.
  • Return the sum of the weighted gaps as a float. Skip empty bins.
The idea behind it

A model is calibrated when its probabilities mean what they say: of all the orders it gives 0.8, about 80 percent are late. Accuracy and AUC do not measure this. A model can rank orders perfectly and still say 0.99 when the truth is 0.6. Calibration matters whenever someone acts on the number itself, such as staffing, pricing or medical risk. Expected calibration error (ECE) gives a single score. Split [0, 1] into equal-width bins. In each bin compare the mean predicted probability, the confidence, with the observed share of positives, the accuracy of the bin. ECE is the average absolute gap, weighted by how many predictions fall in each bin. Zero means perfectly calibrated on this data. ECE depends on the number of bins and gets noisy when bins hold few examples, so teams report the bin count with it.

pythonCtrl+Enter runs
import numpy as np

def expected_calibration_error(probs, labels, n_bins):
"""Expected calibration error over equal-width bins.

probs: list of n floats in [0, 1], predicted chance of a late order
labels: list of n ints, 1 if the order was late and 0 otherwise
n_bins: number of equal-width bins on [0, 1]
Returns a float: the sum over non-empty bins of
(bin count / n) * |share late in bin - mean prob in bin|.
A probability p goes to bin min(floor(p * n_bins), n_bins - 1).
"""
raise NotImplementedError

Stuck? Reveal one rung at a time

Worked solution

Pro includes the solution and walkthrough for every problem, plus the hands-on labs. The first problem of each topic is free.

Get Pro Sign in