Synthetic Data Generation for Model Training

Hiring a team of human annotators to write 52,000 instruction-response pairs would cost ~$500,000 and take months. In March 2023, Stanford's Alpaca team did it for $500 in OpenAI API credits, in about 3 hours, by having text-davinci-003 generate the data itself. That moment is when instruction-tuning stopped being a moat.

The technique — Self-Instruct (Wang et al., 2022) — has since become the default recipe. Every open instruction dataset you've heard of (Alpaca, Dolly, WizardLM, UltraChat, OpenOrca) is built on it. Microsoft's Phi-3 technical report goes further: Phi-3 matches GPT-3.5's benchmarks with 3.8B parameters largely because ~70% of its training data is LLM-generated, curated, "textbook-quality" synthetic text.

The core insight: a capable LLM already knows enough to write training data for a smaller/weaker model. You just have to:

1. Write a handful of diverse seed instructions yourself
2. Prompt the LLM to generate more in the same style
3. Parse, filter, deduplicate — then train on the result

What you'll build

A miniature Self-Instruct pipeline end-to-end on your GPU:

1. Seeds + generator — load TinyLlama-1.1B-Chat, pick 5 seed instructions
2. Generate + parse — produce raw completions, parse structured instructions out of them
3. Quality filter — length gates and format checks
4. Dedup + diversity — remove near-dupes, measure diversity, produce the final dataset

We'll use TinyLlama-1.1B-Chat as the generator — 1.1B params, Apache-2.0 license, no gating, ~2 GB VRAM. Alpaca used GPT-3.5 to generate training data for a 7B Llama; we use a 1.1B to generate data for illustration. The pipeline is identical.

Two references to read while this runs

• Self-Instruct: Aligning Language Models with Self-Generated Instructions — Wang et al., 2022. The paper that started the line. Sections 2 and 3 are the recipe.
• Stanford Alpaca README — the canonical productionized implementation, under ~500 lines of Python. Read generate_instruction.py.

---

Self-Instruct starts with a tiny human-written seed set. The original paper used 175 carefully diverse seeds; Alpaca cut it to a smaller manually-curated set. The seeds define the shape and breadth of what the generator will imitate.

Seed quality matters enormously — if your seeds are all coding questions, you'll get coding questions back. Common practice: vary across:

• Topic (math, writing, summarization, reasoning, creative, factual)
• Style (command, question, roleplay, constraint)
• Length (one-liner, multi-sentence, with examples)

seed_instructions = [
    {'instruction': 'Explain Bayes theorem in two sentences.',
     'output': 'Bayes theorem tells you how to update a belief when you see new evidence. The updated probability is proportional to how likely the evidence was under that belief times your prior estimate of the belief.'},
    {'instruction': 'Write a SQL query that returns the top 3 salaries from a table called employees.',
     'output': 'SELECT salary FROM employees ORDER BY salary DESC LIMIT 3;'},
    {'instruction': 'Summarize the plot of Pride and Prejudice in one paragraph.',
     'output': 'Elizabeth Bennet, the sharp-witted second of five sisters, initially despises the aloof Mr. Darcy, but as she uncovers his hidden generosity and reevaluates her own prejudices, the two overcome the social gap between them and marry.'},
    {'instruction': 'Give me three reasons to use type hints in Python.',
     'output': '1. Catch bugs earlier via static checking with mypy. 2. Make function contracts self-documenting. 3. Enable better IDE autocomplete and refactoring.'},
    {'instruction': 'What is a closure in programming?',
     'output': 'A closure is a function that captures variables from its enclosing scope and carries them with it, letting the function reference those variables even after the enclosing scope has exited.'},
]

print(f'{len(seed_instructions)} diverse seed instructions ready')
for s in seed_instructions:
    print(f"  - {s['instruction']}")

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = 'TinyLlama/TinyLlama-1.1B-Chat-v1.0'

gen_tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
gen_model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.float16).to('cuda')
gen_model.eval()

print(f'Loaded {MODEL_ID}')
print(f'  Parameters: {gen_model.num_parameters() / 1e9:.2f}B')
print(f'  VRAM used by weights: ~{gen_model.num_parameters() * 2 / 1e9:.2f} GB (fp16)')
print(f'  Free VRAM: {torch.cuda.mem_get_info()[0] / 1e9:.2f} GB')