Preporato

Gradient descent

The update rule behind every trained model, and how it goes wrong.

Start 5 problems

Problems

  1. 1One gradient descent stepEasy · 8 min · 10 hidden testsSolution free
  2. 2Mini-batches in a given orderEasy · 10 min · 9 hidden tests
  3. 3Momentum on a lopsided bowlMedium · 14 min · 11 hidden tests
  4. 4Learning rate with warmup and cosine decayMedium · 12 min · 12 hidden tests
  5. 5Find where a training run divergedMedium · 12 min · 13 hidden tests

Practice it at full scale

Hands-on labs on real tooling, in your browser.

A PyTorch Training Loop From ScratchDebug a PyTorch Training Run