Data Preparation for LLM Training

The unglamorous truth: ~90% of the work to fine-tune a good LLM is data work. Only ~10% is training. If your data is dirty, noisy, or duplicated, no amount of GPU time or clever loss functions will save you.

Two research findings to anchor everything we'll do:

• Lee et al., 2022 — Deduplicating Training Data Makes Language Models Better. Found that existing training datasets contain many near-duplicate examples and long repetitive substrings. After deduplication, models emit memorized text 10× less frequently and reach the same accuracy in fewer steps. Meta, Google, and OpenAI now run aggressive dedup as a standard preprocessing step.
• Zhou et al., 2023 — LIMA: Less Is More for Alignment. Took LLaMA-65B and fine-tuned on just 1,000 carefully curated instruction-response pairs. Human evaluators preferred LIMA's outputs over GPT-4's in 43% of cases. The takeaway: diversity and quality > quantity. Scaling noisy data does almost nothing for alignment.

Context for scale: Llama 3 was trained on ~15 trillion tokens, which is roughly 5× more than Llama 2. Meta reports that the data pipeline — filtering, dedup, categorization — made a larger quality difference than the model architecture changes. See the Llama 3 blog post.

---

What you'll build in this lab

A real preprocessing pipeline you could apply to a scraped corpus before fine-tuning:

1. Load + explore — You can't clean what you haven't looked at.
2. Quality filter — Cheap heuristics that remove the worst 20–40% of samples.
3. Deduplication — Exact-hash dedup. We'll demonstrate a subtle normalization bug and let you fix it.
4. Train a BPE tokenizer — The same algorithm GPT-2, GPT-4, and Llama 3 all use. You'll implement one piece yourself.
5. Format for instruction tuning + validate on GPU — Assemble a batch, push to CUDA, run a real forward pass.

Throughout you'll see # 🐛 Common mistake cells. Read them — they teach the sharp edges of data prep that will absolutely bite you in a real fine-tuning project.

---

Golden rule of data work: look before you filter. Running filters without knowing your data's shape is how you accidentally discard 95% of your corpus because of an invisible assumption (wrong encoding, unexpected separators, language you didn't know was in there).

We'll use wikitext-2-raw-v1 as a stand-in for a collected corpus. Real-world scrapes (CommonCrawl, Dolma, RedPajama) have the same shape problems at a much larger scale. See the HuggingFace datasets hub for the public corpora everyone starts from.

Do: Load the dataset and build raw_texts — a list of non-empty strings from the text column. Then look at the length distribution before doing anything else.

from datasets import load_dataset

ds = load_dataset('Salesforce/wikitext', 'wikitext-2-raw-v1', split='train')
raw_texts = [t for t in ds['text'] if t.strip()]

print(f'Raw corpus: {len(raw_texts):,} non-empty lines')
print(f'Example line: {raw_texts[10][:200]!r}')

# Length distribution — is the data mostly headers, mostly paragraphs, or a mix?
import statistics
lengths = [len(t) for t in raw_texts]
print(f'min: {min(lengths)}  |  p50: {int(statistics.median(lengths))}  |  p95: {int(sorted(lengths)[int(len(lengths)*0.95)])}  |  max: {max(lengths)}')
print(f'Lines under 50 chars: {sum(1 for L in lengths if L < 50):,} ({100*sum(1 for L in lengths if L < 50)/len(lengths):.1f}%)')
print(f'Lines over 5000 chars: {sum(1 for L in lengths if L > 5000):,}')