PreporatoPreporato
Synthetic Data Generation for Model Training
GPU sandbox · jupyter
Beta

Synthetic Data Generation for Model Training

Build a Self-Instruct style synthetic dataset end-to-end: seed instructions, LLM-driven generation, robust parsing, quality filtering, and dedup + diversity scoring. The same pipeline that produced Alpaca, WizardLM, and most modern instruction-tuning corpora.

40 min·4 steps·3 domains·Intermediate·ncp-genlncp-adsnca-genl

What you'll learn

  1. 1
    Seed instructions + load the generator
  2. 2
    Generate & parse
  3. 3
    Quality filter
  4. 4
    Dedup + diversity + final dataset

Prerequisites

  • Familiar with Hugging Face transformers and tokenizers
  • Comfortable with Python list/dict data wrangling
  • Basic understanding of instruction tuning

Exam domains covered

Data Analysis and VisualizationExperimentationLLM Integration and Development

Skills & technologies you'll practice

This intermediate-level gpu lab gives you real-world reps across:

Synthetic DataSelf-InstructInstruction TuningData QualityDeduplicationTinyLlamaDataset Curation

What you'll build in this synthetic-data lab

Synthetic data is the only reason frontier fine-tuning is tractable in 2026 — Alpaca, WizardLM, Orca, every domain instruction corpus that isn't scraped off the public web, and most of what powers open-weight instruction tunes came out of a Self-Instruct-style pipeline. In about 40 minutes on a real NVIDIA GPU we provision, you'll implement that pipeline end-to-end: seed examples, LLM-driven generation with sampling, robust parsing of free-form LM output, length + repetition filters, exact-hash dedup, and a diversity score you can watch collapse if your seeds are too homogeneous. You'll walk away with a working mental model of synthetic data as a funnel (not a fountain) and concrete numbers for how many samples die at each stage — parse failures alone typically kill 30-60% on first pass.

The substance is the scaffolding you'll end up rewriting whenever you generate training data for a real fine-tune. You'll build a few-shot prompt from hand-crafted seeds, call model.generate(do_sample=True, temperature=0.7-1.0) — high enough to get instruction diversity, high enough that your regex parser will fail on some outputs — and watch parse-success rates in the 40-70% band that production Self-Instruct pipelines actually hit. You'll apply length gates ([10, 500] chars for instructions, [10, 2000] for outputs), strip degenerate repetition, then normalise and exact-hash dedup with a set. The critical lesson is that exact-hash dedup is a floor: at 50k samples you'll have thousands of templated paraphrases ("Write a haiku about the sea" vs "Compose a haiku about the ocean") that lex-differ but teach the model the same thing, which is why production pipelines layer embedding-based semantic dedup on top. TinyLlama is the generator here because it iterates in seconds — swap in Llama 3 70B, Mixtral, or a frontier API model for production and the pipeline code doesn't change.

Prerequisites: comfort with Hugging Face transformers, Python list/dict wrangling, and the basic shape of instruction tuning. TinyLlama weights and the tokenizer are cached in the sandbox. The grader is deliberately realistic about stochastic generation — if a run produces zero quality-passing samples, it surfaces the diagnostic rather than failing silently, because that outcome is itself a teaching moment about generator choice. Search-intent hooks: "generate synthetic training data for LLM", "Self-Instruct pipeline tutorial", "Alpaca-style data generation", "deduplicate instruction tuning dataset".

Frequently asked questions

Why use TinyLlama specifically — won't a stronger generator produce better data?

Absolutely it would, and that's one of the reflection lessons. TinyLlama is used here because it fits on a single small GPU and runs fast enough that you can iterate on prompts and filters in seconds. In production you'd swap in Llama 3 70B, Mixtral, or even a frontier API model — the pipeline code doesn't change, only the generator handle does. The point of the lab is the scaffolding, not the generator choice.

Why expect parse-success well below 100%?

Because LLM output is fundamentally free-form and your regex/format parser is strict. A sampling temperature of 0.7-1.0 is what gives you instruction diversity, but it also makes the model occasionally skip delimiters, hallucinate extra fields, or emit markdown where you wanted plain text. Production Self-Instruct pipelines hit 40-70% parse success on first pass and compensate by generating more — tightening the parser below that level drops diversity faster than it drops noise.

What's wrong with exact-match deduplication at scale?

It only catches byte-identical strings after normalization. A 50k dataset generated from similar prompts fills up with templated rewrites — 'Write a haiku about the sea', 'Compose a haiku about the ocean', 'Write me a haiku about a sea' — that lexically differ but teach the model the same lesson ten thousand times. Embedding-based dedup (compute BGE vectors, cluster, keep one per cluster) is what catches those, and it's the first upgrade the reflection step asks you to defend.

How should I pick a diversity score — token ratio, edit distance, or verb coverage?

All three are valid for this lab as long as they land in [0, 1]. Unique-token ratio (unique tokens / total tokens across all instructions) is cheapest and catches degenerate repetition well. Average pairwise edit distance catches near-duplicates that token ratio misses. Verb coverage (count of distinct lead verbs — 'write', 'classify', 'summarize', etc.) is what the original Self-Instruct paper used because it approximates task-type diversity. In practice you'd track all three and alert when any one collapses.

Why do seed examples matter so much when the generator produces most of the samples?

Because the few-shot prompt is what defines the task distribution the model generates into. Three seeds that are all 'write a haiku about X' collapse the output to haikus. Three seeds covering classification, extraction, and open-ended writing produce a much broader distribution. This is why the original Self-Instruct used 175 hand-written seeds and Alpaca used 175 as well — the seeds are the implicit task ontology, and they're worth hand-crafting.

When should I use LLM-as-judge filtering vs regex rules?

Length and format rules catch obvious junk for almost no cost — run them first. LLM-as-judge catches what rules can't see: factual errors, failure to follow the instruction, subtle toxicity, low-effort completions. The cost is real, though — every kept sample means an extra inference call, and the judge has its own biases (favoring verbose answers, penalizing correct-but-brief ones). The right pattern is regex rules as a cheap first pass, LLM-as-judge on the survivors, and human review on a calibration set that tunes both.