Preporato

Attention and transformers

From attention to a token: the parts of a GPT, one function at a time.

Start 6 problems

Problems

  1. 1The additive causal maskEasy · 8 min · 10 hidden testsSolution free
  2. 2Sinusoidal position encodingsEasy · 10 min · 10 hidden tests
  3. 3Scaled dot-product attention with a maskMedium · 15 min · 11 hidden tests
  4. 4Split into heads, gate, merge backMedium · 15 min · 10 hidden tests
  5. 5Temperature and top-p filteringMedium · 15 min · 12 hidden tests
  6. 6One decode step with a KV cacheHard · 25 min · 10 hidden tests

Practice it at full scale

Hands-on labs on real tooling, in your browser.

Build a BPE Tokenizer From Scratch in Python (and See Why Some Languages Cost More)Build a Transformer from ScratchBuild a Tiny GPT From Scratch on a CPU

Where interviews test it

Cases from our interview prep that lean on this step.

Fix the attention layer, then add a KV cache A reproducible sampler with top-k, top-p and speculative acceptance