Sigmoid that never overflows
Easy10 min10 hidden tests
A click model turns raw scores into probabilities with the sigmoid, 1 / (1 + exp(-z)), and backprop needs its slope too. Some raw scores reach 1000 or -1000. There exp overflows to infinity, and common ways of writing the sigmoid or its slope return nan (not a number). Write sigmoid_and_grad(z) that returns the sigmoid of every score and its derivative, both finite for any finite score.
Drag z, or try a huge score
s = 1 / (1 + e^−z), ds = s(1 − s)z ≥ 0, so use 1 / (1 + e^−z): e^−z = 0.1353. s = 0.8808, slope s(1 − s) = 0.105. The exp only ever sees a number 0 or below, so it stays at most 1.
z 2. Sigmoid 0.8808, slope 0.105.
Examples
The sigmoid is 0.5 at zero, and its slope peaks there at 0.25
sigmoid_and_grad([-2.0, 0.0, 2.0])
returns ([0.119203, 0.5, 0.880797], [0.104994, 0.25, 0.104994])
Huge scores give probabilities of 1 and 0 with a slope of 0
sigmoid_and_grad([1000.0, -1000.0])
returns ([1, 0], [0, 0])
Requirements
zis a list or 1-D array of finite floats, as large as 1000 or as small as -1000.- Return a tuple
(s, ds)of two 1-D NumPy float arrays with the same length asz.sis the sigmoid anddsis its derivative,s * (1 - s). - Every call to
expmust stay finite, even where an overflow would happen to give the right output. - Neither array may hold
nanor infinity.
The idea behind it
The sigmoid, s(z) = 1 / (1 + exp(-z)), squeezes any real number into the range 0 to 1. Logistic regression and binary classifiers use it to turn a raw score, called a logit, into a probability. Its derivative has a tidy form, s(z) * (1 - s(z)), so backprop can reuse the value it already has. The danger is floating point. A 64-bit float tops out near 1.8e308, so exp(x) overflows to infinity once x passes about 709. A 32-bit float, common in training, overflows past about 88. At z = -1000 the plain form still lands on 0, since 1 / infinity is 0, but only after an overflow warning, and common rewrites of it give nan. The fix is algebra: for negative z, the same sigmoid equals exp(z) / (1 + exp(z)), and there exp(z) is tiny. Choosing the form by the sign of z keeps every exponent at 0 or below, which is why libraries build in this split.