LLM training from scratch labs
Eight GPU labs from a raw corpus to coherent text generated from weights you trained yourself.
Training a language model from scratch means starting from randomly initialized weights: clean a corpus, train a tokenizer, build the transformer, run the pretraining loop, and watch validation perplexity (the exponential of the average per-token loss, so lower means the model is less surprised by held-out text) fall until the model writes coherent text. The models in these labs are small enough to train in one session on one GPU, and the mechanics are exactly what the NVIDIA NCP-GENL Data Preparation and LLM Architecture domains and the NCA-GENL Core Machine Learning and AI Knowledge domain ask about: tokenization and vocabulary, attention and LayerNorm, learning-rate schedules, quality filtering, and what a domain-adaptation run gains and forgets. Those questions get much easier once you have wired the pieces together yourself and seen what each one changes.
- A filtered, deduplicated corpus and your own BPE tokenizer, batch-validated on a GPU
- A Self-Instruct style synthetic instruction dataset with parsing, quality filtering and diversity scoring
- A decoder-only transformer written by hand, and a GPT-style model pretrained from zero on TinyStories that generates coherent text from your own weights
- GPT-2 continued-pretrained on Python code, with the gain on code and the forgetting on English both measured
- Same-seed runs that match bit for bit, an MLflow-tracked run with a registered model and sweep, and a DALI input pipeline benchmarked against the PyTorch DataLoader
All 8 labs run on dedicated NVIDIA GPU pods in the browser. No local GPU and nothing to install.
- PyTorch basics: tensors, a training loop, what a loss curve is
- The idea of a token and a vocabulary; the data lab trains a tokenizer from scratch
- Comfort with matrix multiplication and softmax for the transformer lab; no prior pretraining experience needed
How the path is sequenced
The collection runs in project order. Data first: load a raw corpus, apply quality filters, deduplicate, train a BPE tokenizer (byte-pair encoding, which builds a vocabulary by repeatedly merging the most frequent adjacent pairs) and batch-validate on the GPU; then generate a Self-Instruct style synthetic instruction dataset with LLM-driven generation, robust parsing, quality filtering and dedup plus diversity scoring; then move image decoding, resizing and augmentation onto the GPU with NVIDIA DALI and benchmark it against a standard PyTorch DataLoader. Model next: write scaled dot-product attention, multi-head attention and a full block with residuals and LayerNorm by hand and train the tiny GPT you assembled; pretrain a GPT-style model from zero on TinyStories with your own optimizer and learning-rate schedule, tracking validation perplexity, and generate text from the weights; then continue pretraining GPT-2 on Python code for 150 steps and measure the gain on code against the catastrophic forgetting (lost ability on the original English data). The last two labs make the work repeatable: flip PyTorch determinism flags until same-seed runs match bit for bit and quantify the speed cost, then wire MLflow into a training loop with tracked runs, a registered model, a multi-run sweep and a production spec.
All eight labs run on dedicated NVIDIA GPU pods in the browser, take 30 to 55 minutes each, and are checked step by step.
Feed the model
Filter, deduplicate and tokenize a corpus, generate a synthetic instruction set, and move the input pipeline onto the GPU with DALI.
Build and pretrain
Write the transformer by hand, train a small language model from zero, then domain-adapt GPT-2 and measure what it forgets.
Make it repeatable
Bit-for-bit reproducible runs, then MLflow tracking, a model registry and a sweep on top of the training loop.
Run all 8 labs in this path, and every other lab and practice test on Preporato.
- All 8 labs in this path
- Dedicated GPU pods included
- Every certification practice test
- Cancel any time
Frequently asked questions
Yes, three times. The transformer lab has you write attention, masking and LayerNorm yourself and train the tiny GPT you assembled; the small-language-model lab pretrains a GPT-style model from zero on TinyStories and generates text from your own weights; the continued-pretraining lab runs 150 steps on GPT-2 and measures the trade-off. Each run fits inside a single lab session on one GPU.
Tokenization, vocabulary management, quality filtering, data augmentation and transformer architecture sit in the NVIDIA NCP-GENL Data Preparation and LLM Architecture domains. On NCA-GENL, transformer fundamentals, attention and loss functions are in the largest domain (Core Machine Learning and AI Knowledge), and experiment design and evaluation metrics are in its Experimentation domain. The reproducibility and MLflow labs back the experiment-tracking questions on both exams.
Because a starved input pipeline caps training throughput no matter how good the model code is. The DALI lab moves decoding, resizing and augmentation from CPU to GPU and benchmarks the result against a standard PyTorch DataLoader, which is the clearest way to see a data-loading bottleneck and its fix. DALI operates on images, so that is the workload the lab uses; the diagnosis (is the GPU waiting on data?) is the same for any training job.
The labs are included in Preporato Pro ($29.99 per month or $290 per year), together with every practice test and every other lab on Preporato. The eight labs total a little under six hours (30 to 55 minutes each) and are independent, so you can do the transformer lab alone if that is the piece you need.