Find where a training run diverged
Medium12 min13 hidden tests
An overnight training job logs its loss once per step, and the on-call engineer wants an alarm that names the first step where the run went wrong. Step i diverged when its loss is not a finite number (nan or infinity), or when it is more than factor times the lowest loss seen before step i. Write first_divergence(losses, factor) that returns that step, or -1 when the run stayed healthy.
Drag a loss, change the factor
diverged: nan, inf, or > factor × best so farStep 6 diverged: its loss 2.6 is more than 2.0 × 1.1 = 2.2, the best loss before it times the factor.
First diverged step 6.
Examples
4.0 is more than twice the best so far, 1.5
first_divergence([2.3, 1.9, 1.5, 4.0, 1.2], 2.0)
returns 3
A nan loss counts as diverged
first_divergence([2.3, 1.9, math.nan, 1.0], 2.0)
returns 2
Requirements
lossesis a list of floats, possibly empty. Finite losses are positive, and some entries may benanorinf.factoris a float above 1.- Step i diverged when
losses[i]isnanor infinite, or whenlosses[i] > factor * min(losses[:i]). Step 0 has no earlier loss, so it diverges only when its loss is not finite. - A loss exactly equal to
factortimes the best so far is still healthy. - Return the first diverged index as an int, or -1 when no step diverged.
The idea behind it
A training run diverges when its loss stops falling and shoots up, usually because the learning rate is too high, a batch holds bad data, or a value overflows. Sometimes the loss turns into nan (not a number), and from then on every weight it touches becomes nan too. Catching the first bad step early saves compute and points to the cause: the batch, the learning rate at that step, or a gradient spike. A useful rule compares each loss with the best loss so far. Losses are noisy, so a small rise from one step to the next is normal, and comparing with the previous step either fires too often or misses a slow climb. A ratio against the running minimum ignores noise and still catches a real blowup. The test for nan must be explicit, because in floating point any comparison with nan is False.