Optimizers and normalization
Adam, weight decay, clipping and norms: what keeps training stable.
Start 5 problems
Problems
- 1Clip gradients by their global normEasy · 10 min · 11 hidden testsSolution free
- 2One Adam step with bias correctionMedium · 14 min · 10 hidden tests
- 3AdamW against L2 in the gradientMedium · 15 min · 10 hidden tests
- 4Layer normalization over the last axisEasy · 9 min · 11 hidden tests
- 5Batch norm in training modeMedium · 14 min · 10 hidden tests
Practice it at full scale
Hands-on labs on real tooling, in your browser.
