Inverted dropout, training and eval
Easy8 min11 hidden tests
A speech model uses dropout on its hidden activations: during training, each unit is zeroed with probability p. To keep the test reproducible, the random draw is given to you as keep_mask, with 1 for a kept unit and 0 for a dropped one. Write dropout(x, keep_mask, p, training) using inverted dropout: in training, zero the dropped units and divide the kept ones by 1 - p. In eval mode, return the activations unchanged. With x = [1.0, 2.0, 3.0, 4.0], mask [1, 0, 1, 1] and p = 0.25, training gives about [1.333, 0.0, 4.0, 5.333].
Tap a unit to drop or keep it
training: x · mask / (1 − p); eval: xTraining: x × mask / (1 − 0.25) = [1.333, 0, 4, 5.333]. Kept units grow by 1 / 0.75 = 1.333, so on average over random masks the total stays where it is in eval (10); this mask gives 10.667.
Output 1.333, 0, 4, 5.333.
Examples
Training: one unit dropped, the rest scaled up by 1 / 0.75
dropout([1.0, 2.0, 3.0, 4.0], [1, 0, 1, 1], 0.25, True)
returns [1.33333, 0, 4, 5.33333]
Eval: the mask is ignored
dropout([1.0, 2.0, 3.0, 4.0], [1, 0, 1, 1], 0.25, False)
returns [1, 2, 3, 4]
Requirements
xandkeep_maskare lists or arrays of the same shape (1-D or 2-D).keep_maskholds 0s and 1s.pis the drop probability, with 0 <= p < 1.trainingis True or False.- Training: return
x * keep_mask / (1 - p). Eval: returnxas it is and ignore the mask. - Return a float NumPy array with the same shape as
x.
The idea behind it
Dropout is a regularizer: during training it zeroes each unit at random with probability p, so the network cannot lean on any single unit. That changes the scale of a layer's output. If 25% of units vanish, the next layer sees about 75% of the usual signal. Inverted dropout fixes the scale during training. It divides the kept units by the keep rate 1 - p, so each unit's expected value stays equal to its input: E[x * mask / (1 - p)] = x * (1 - p) / (1 - p) = x. At eval time, dropout does nothing at all. This is why PyTorch has model.train() and model.eval(): forgetting to switch to eval leaves dropout on and makes predictions noisy. A mask passed in from outside, as here, is also how you make a dropout test repeatable.