How sure is an eval score

Easy8 min10 hidden tests

An evaluation team grades a chatbot on a test set: each question gets a score from 0 to 1, where 1.0 is fully right and 0.0 is wrong. Before anyone claims a new version is better, they want to know how much the average would move on a different test set of the same size. Write metric_summary(scores) that returns the mean score, the sample standard deviation of the scores and the standard error of the mean.

Grade it on another test set

standard error = s / √n

Right (1)Wrong (0)

Questions in the test set25

Mean 0.60, sample std 0.50, standard error 0.100 = 0.50 / √25. A new set of 25 questions usually lands within two standard errors (±0.20) of the true score.

25 questions. Mean 0.60, standard error 0.100.

Examples

Three of four answers right: mean 0.75, std 0.5, standard error 0.25

metric_summary([1, 0, 1, 1])

returns (0.75, 0.5, 0.25)

Partial grades work the same way

metric_summary([0.8, 0.6, 0.7, 0.9, 1.0])

returns (0.8, 0.158114, 0.0707107)

Requirements

scores is a list or 1-D array of n numbers, with n of 2 or more.
The sample standard deviation divides the sum of squared gaps from the mean by n - 1, then takes the square root.
The standard error is the sample standard deviation divided by the square root of n.
Return a tuple (mean, std, standard_error) of three floats.

The idea behind it

A test set is a sample of the questions a model will face. Run the same model on a different sample of the same size and the average score moves. The standard error of the mean measures how much:

SE = s / sqrt(n)

Here s is the sample standard deviation of the per-example scores. It divides by n - 1 (Bessel's correction) because the gaps are measured from the sample's own mean, which sits a little closer to the data than the true mean does.

A rough 95% interval is the mean plus or minus 1.96 standard errors. With 0 or 1 scores, 200 questions and 90% accuracy, the standard error is about 2.1 points, so the interval is about plus or minus 4.2 points. Two versions 2 points apart on that test set are not clearly different. Halving the standard error takes four times as many examples, which is why careful eval teams size their test sets before they compare models.

Tested in interview case Readiness check for the AI Solutions Architect loop.

pythonCtrl+Enter runs

import numpy as np

def metric_summary(scores):

"""Mean, sample standard deviation and standard error of eval scores.

scores: list or 1-D array of n per-example scores, n >= 2

Returns (mean, std, standard_error) as three floats, where std is the

sample standard deviation (divide by n - 1) and

standard_error = std / sqrt(n).

"""

raise NotImplementedError

Stuck? Reveal one rung at a time

Worked solution

Give it a real try first. Then read the solution and why each line is there.