Preporato

Gradient check for a backward pass

Medium12 min12 hidden tests

A team hand-wrote the backward pass for a new layer. Before they train with it, they compare its gradient with a numerical estimate from finite differences. Write gradient_check(analytic, numeric, tol) that returns the relative error between the two gradients and whether it is small enough to trust. The relative error is norm(a - b) / (norm(a) + norm(b)), where norm is the Euclidean length.

Drag the numeric gradient

error = |a − b| / (|a| + |b|)
Analytic aNumeric bGap a − b

|a − b| = 1.41e−6 and |a| + |b| = 4.4721, so the relative error is 3.16e−7. That is at most tol 1e−6: ok, trust the backward pass.

Relative error 3.16e−7, ok.

Examples

A tiny gap relative to the gradient's size passes

gradient_check([1.0, -2.0], [1.000001, -2.000001])

returns (3.16228e-07, True)

A flipped sign gives a large error and fails

gradient_check([1.0, -2.0], [1.0, 2.0])

returns (0.894427, False)

Requirements

  • analytic and numeric are lists or arrays of the same shape, 1-D or 2-D. norm treats all entries of an array as one long vector: the square root of the sum of their squares.
  • When both gradients are all zeros, they agree exactly and the error is 0.0.
  • Return a tuple (error, ok): error is a float, and ok is True when error is at most tol. The default tol is 1e-6.
The idea behind it

A gradient check compares the gradient from your backward pass with an estimate from finite differences: nudge each input, see how the loss moves. The two never match exactly, because the estimate has rounding and truncation error, so you need a measure of how far apart they are. An absolute gap misleads, since gradients can be 1e-6 in one layer and 1e3 in another. The relative error ||a - b|| / (||a|| + ||b||) divides by the size of both, so it falls between 0 and 1: 0 for a perfect match, 1 when one of them is zero and the other is not. Adding both norms in the denominator keeps it defined when either gradient alone is zero. As a rough guide in 64-bit floats with a central difference, errors near 1e-7 or below usually mean the backward pass is right, and errors near 1e-2 or above usually mean a bug.

pythonCtrl+Enter runs
import numpy as np

def gradient_check(analytic, numeric, tol=1e-6):
"""Relative error between two gradients, and whether it passes.

analytic: list or array, the gradient from the backward pass
numeric: list or array of the same shape, the finite-difference estimate
tol: the largest error that still passes, a positive float
Returns (error, ok): a float and a bool.
"""
raise NotImplementedError

Stuck? Reveal one rung at a time

Worked solution

Pro includes the solution and walkthrough for every problem, plus the hands-on labs. The first problem of each topic is free.

Get Pro Sign in