Reproducible Training: The Flags, The Cost, The Artifacts

Before we prove determinism works, we need to see the cost of not having it. Two runs with different seeds — the exact scenario when someone "forgot to set the seed" or changed an unrelated library version that reshuffled internal RNG calls — diverge noticeably:

• Final training loss varies
• Weights diverge in L2

Why does that matter? If the noise floor from seed variation is bigger than the improvement from your PR, you can't claim the PR helped. CI tests that gate on loss < expected + ε either fail randomly (too tight) or pass anything (too loose). Determinism collapses that floor to zero — an improvement of any size is provable.

The model we'll use to measure reproducibility

A tiny embedding+MLP classifier is enough to surface CUDA non-determinism — we don't need a 1B-param model to see floating-point order effects. The train_one_epoch function below is deliberately self-contained: seed → build model → seed data → train N steps → return final loss + a flattened weight tensor. The weight tensor is the key: two bitwise-identical runs produce identical weights, and any deviation shows up as (wa - wb).norm() > 0.

import torch, torch.nn as nn, random, numpy as np

device = 'cuda'

def seed_all(seed=42):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

class EmbedMLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.emb = nn.Embedding(512, 128)
        self.net = nn.Sequential(
            nn.Linear(128, 256), nn.LayerNorm(256), nn.GELU(),
            nn.Linear(256, 256), nn.LayerNorm(256), nn.GELU(),
            nn.Linear(256, 10),
        )
    def forward(self, idx):
        return self.net(self.emb(idx).sum(dim=1))

def train_one_epoch(seed=42, n_steps=200):
    seed_all(seed)
    model = EmbedMLP().to(device)
    opt = torch.optim.AdamW(model.parameters(), lr=1e-3)
    loss_fn = nn.CrossEntropyLoss()
    torch.manual_seed(seed)
    n_samples = 2048
    data_idx = torch.randint(0, 512, (n_samples, 32), device=device)
    data_y   = torch.randint(0, 10, (n_samples,), device=device)
    final_loss = None
    for step in range(n_steps):
        pick = torch.randint(0, n_samples, (64,), device=device)
        x, y = data_idx[pick], data_y[pick]
        opt.zero_grad()
        loss = loss_fn(model(x), y)
        loss.backward()
        opt.step()
        final_loss = loss.item()
    weights = torch.cat([p.detach().flatten() for p in model.parameters()])
    return final_loss, weights

The noise floor — two runs with determinism OFF

Before fixing anything, measure what you're fighting against. Below we run train_one_epoch twice with different seeds (42 vs 43) and PyTorch defaults (cudnn.benchmark=True, no deterministic algorithms). This simulates the real-world failure mode: a stray seed change upstream, or a dataloader that seeds itself per-worker and forgot to inherit the main seed.

The weight L2 diff you get is the noise floor — if you try to measure a 5% loss improvement on a PR but this diff is >1, your PR's improvement is lost in the seed variance. That's why the determinism flags in step 2 matter.

# Default PyTorch — not deterministic by guarantee.
torch.use_deterministic_algorithms(False)
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.benchmark = True

# Two different seeds — simulates the scenario where someone forgot to set the seed
# (so RNG state differs run-to-run) or the seed was plumbed wrong somewhere upstream.
# This is the *noise floor* you're fighting against.
nondet_run_a_final_loss, wa = train_one_epoch(seed=42)
nondet_run_b_final_loss, wb = train_one_epoch(seed=43)
nondet_weight_diff = (wa - wb).norm().item()

print(f"Two runs, seeds 42 vs 43 (simulating a stray seed change):")
print(f"  run A (seed=42) final loss: {nondet_run_a_final_loss:.6f}")
print(f"  run B (seed=43) final loss: {nondet_run_b_final_loss:.6f}")
print(f"  |loss_a - loss_b| = {abs(nondet_run_a_final_loss - nondet_run_b_final_loss):.3e}")
print(f"  weight L2 diff:     {nondet_weight_diff:.3e}")
print()
print("That gap is the noise floor you'd be fighting on every PR.")
print("A PR that improves loss by 5% is indistinguishable from this seed variance.")
print("Step 2 shows what happens when you pin seed + turn on determinism flags.")