Bootstrap interval for a mean
Medium14 min10 hidden tests
An eval team wants an interval for a model's mean score without assuming the scores follow a bell curve. They bootstrap: build many resamples of the test set by drawing examples with replacement, take the mean of each resample, and read percentiles off those means. To keep reports reproducible, the resample indices are drawn once and passed in. Write bootstrap_interval(values, resamples, level) that returns the percentile bootstrap interval.
Add resamples, change the level
interval = percentiles of the resample means5 resample means, sorted: 3, 5, 5, 6.5, 7.5. Percentiles 25 and 75 sit at positions 1 and 3 of that list, read with linear interpolation: 5 to 6.5.
5 resamples. Interval 5 to 6.5.
Examples
Five resample means, 3, 5, 7.5, 5 and 6.5, give a 50% interval from 5 to 6.5
bootstrap_interval([2.0, 4.0, 6.0, 8.0], [[0, 0, 1, 1], [0, 1, 2, 3], [2, 3, 3, 3], [1, 1, 2, 2], [3, 3, 3, 0]], 0.5)
returns (5, 6.5)
Eight resamples of six pass or fail scores
bootstrap_interval([0.0, 1.0, 1.0, 0.0, 1.0, 1.0], (np.arange(48).reshape(8, 6) ** 2 // 7) % 6, 0.8)
returns (0.283333, 0.833333)
Requirements
valuesis a list or 1-D array of n scores.resamplesis a 2-D list or array of integers with shape (B, n): row b lists the indices ofvaluesin resample b, with repeats allowed.levelis a float between 0 and 1, such as 0.9.- Compute B resample means, one per row.
- Return
(low, high): the percentiles 100 * (1 - level) / 2 and 100 * (1 + level) / 2 of those B means, as two floats. - Use linear interpolation, NumPy's default: sort the means, find position (B - 1) * q / 100 for percentile q, and interpolate between the two nearest sorted means.
The idea behind it
The bootstrap estimates uncertainty by treating the sample as a stand-in for the population. Draw n examples from the test set with replacement, so some appear twice and some not at all, and compute the statistic on that resample. Repeat B times. The spread of the B results shows how much the statistic would move on a fresh test set.
The percentile interval reads the answer straight off the sorted results. For a 90% interval, cut 5% from each end: the 5th and 95th percentiles of the resample means. It needs no formula for the standard error and no bell-curve assumption, so it works for medians, F1 scores and other metrics that have no simple formula.
In practice B is 1,000 or more, and the indices come from a seeded random generator so the interval is reproducible. Passing the indices in, as here, makes the same idea testable. Percentile definitions vary between libraries; NumPy's default interpolates linearly between the two nearest sorted values.