Xavier and He initialization scales
Easy7 min9 hidden tests
Before training a digit classifier, a team draws each layer's starting weights from a normal distribution with mean 0. The standard deviation decides whether signals shrink, hold or explode as they pass through the layers. Write init_stds(fan_in, fan_out) that returns the standard deviations two common schemes use for a layer with fan_in inputs and fan_out outputs: Xavier (also called Glorot), sqrt(2 / (fan_in + fan_out)), and He, sqrt(2 / fan_in). A layer from 784 pixels to 256 units gets about 0.0439 and 0.0505.
Change the layer’s fans
Xavier √(2 / (in + out)), He √(2 / in)Xavier √(2 / 1040) = 0.0439, He √(2 / 784) = 0.0505. Through 10 ReLU layers with 784 inputs each, He keeps the spread at 1, Xavier ends at 0.24 and a fixed std of 0.1 at over 100.
Xavier 0.0439, He 0.0505.
Examples
A first layer for 28 by 28 pixel digits
init_stds(784, 256)
returns {'xavier': 0.0438529, 'he': 0.0505076}
With equal fans, Xavier is sqrt(1 / 100) = 0.1
init_stds(100, 100)
returns {'xavier': 0.1, 'he': 0.141421}
Requirements
fan_inandfan_outare positive integers: the layer's number of inputs and outputs.- Xavier std is
sqrt(2 / (fan_in + fan_out)). He std issqrt(2 / fan_in). - Return a dict
{"xavier": float, "he": float}holding standard deviations (square roots of the variances).
The idea behind it
A unit in a dense layer sums fan_in weighted inputs. If the weights have variance s^2 and the inputs have variance 1, the sum has variance about fan_in * s^2. Pick s too large and activations grow layer after layer until they overflow. Pick it too small and they fade toward 0, and so do the gradients. Xavier (Glorot) initialization sets the variance to 2 / (fan_in + fan_out), a compromise that keeps the forward signal and the backward gradients both near their input scale. It assumes an activation that is roughly linear near 0, such as tanh. He (Kaiming) initialization sets it to 2 / fan_in. ReLU zeroes about half its inputs, which halves the variance, and the 2 puts it back. For a convolution, fan_in is input channels times kernel height times kernel width.