Step 1: Perplexity: the branching-factor intuition
"If you can't measure it, you can't improve it." — Peter Drucker
Evaluating an LLM is one of the hardest problems in the field. Humans disagree on what "good" means. Automated metrics miss meaning. Running full benchmarks costs hundreds of dollars per model. And if you can't trust your eval, every optimization you do is flying blind.
This lab builds up from the simplest metrics to modern practice:
- Perplexity — the classic, still-useful LM metric. Computed on GPU.
- BLEU / ROUGE — n-gram overlap metrics from the machine-translation / summarization era. We'll show why they fail on LLM outputs, with a concrete example.
- Side-by-side model comparison — generate from two models of different sizes, compare qualitatively.
- LLM-as-a-Judge — the modern pattern. We'll implement the core pattern and detect the position bias that makes naive LLM judging untrustworthy.
Two papers worth reading while this runs
- Zheng et al. 2023 — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. The definitive paper establishing LLM-as-judge as a credible eval method. Shows GPT-4 agrees with human preferences >80% of the time — as high as humans agree with each other. Also enumerates the three main biases (position, verbosity, self-enhancement) and mitigations. Chatbot Arena, the live leaderboard at lmarena.ai, is the practical output of this work.
- HuggingFace — Perplexity of fixed-length models. The canonical reference for computing perplexity correctly on modern transformers (stride, context window, why
model.eval()matters).
The two models we'll compare
gpt2(124M params, 2019) — small but sharp baselinegpt2-medium(355M params, 2019) — ~3× bigger, same era
Both fit comfortably in 8GB VRAM together, and the size effect on quality is visible even on ~5 prompts.
Perplexity is the exponential of cross-entropy loss. It answers: "at each token the model predicts, how many equally-likely options is it choosing between, on average?"
Concrete numbers:
- Perplexity 2 → the model is effectively flipping a coin. Very confident.
- Perplexity 50 → choosing between ~50 words. Typical for a decent small LM on natural text.
- Perplexity 10,000 → basically random over a ~10K vocab. Garbage model.
Perplexity is a relative metric. Comparing perplexities across different tokenizers is meaningless (different vocab sizes change the scale). But on the same tokenizer + same corpus, lower is better.
It has fallen somewhat out of fashion in the LLM era — perplexity doesn't capture reasoning, factuality, or instruction-following. But it's still the first thing you'd sanity-check during pretraining: a sudden perplexity spike usually means your data pipeline broke.
🐛 Common mistake: forgetting model.eval()
If you compute perplexity on a model still in training mode, dropout layers are active, so the forward pass randomly zeros out activations. Your measured perplexity will be higher than it should, and — worse — non-deterministic. If you see different perplexity every run for the same data, this is almost always why.
Do: Load gpt2, move to CUDA, set to eval mode, and compute perplexity over a short evaluation text using the standard exp(mean(loss)) formula. Assign the scalar to ppl.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
device = torch.device('cuda')
tok = AutoTokenizer.from_pretrained('gpt2')
model = AutoModelForCausalLM.from_pretrained('gpt2').to(device)
model.eval() # <— critical: disables dropout, makes perplexity reproducible
eval_text = (
"The quick brown fox jumps over the lazy dog. Machine learning models "
"have become remarkably capable in recent years. Fine-tuning on "
"domain-specific data often produces better results than prompting alone."
)
ids = tok(eval_text, return_tensors='pt').input_ids.to(device)
with torch.no_grad():
out = model(ids, labels=ids)
ppl = torch.exp(out.loss).item()
print(f'GPT-2 perplexity on our eval text: {ppl:.2f}')
print(f'Interpretation: the model behaves as if it had to pick between ~{ppl:.0f} equally-likely next tokens per step.')
