GPU Cost & Efficiency Audit

We'll launch a deliberately bad training loop: tiny model, ~40 ms of pure-Python preprocessing per batch, then forward/backward. This guarantees the GPU spends most of its time waiting for the dataloader. That's the common case in the wild.

The sampler runs in a background thread at 10 Hz while the workload executes in the main thread. We capture: GPU utilization (%), memory used (%), temperature (C), power (W), and a monotonic timestamp.

Why 10 Hz and not faster? pynvml calls take ~1 ms each. 100 Hz is feasible but adds thread-contention noise. 10 Hz is the standard for production monitoring (dcgm-exporter default scrape cadence is also ~1 Hz).

Why a background thread and not nvidia-smi --loop=1? Subprocess + disk parsing each tick is 10× slower than pynvml. And we want the samples in-memory anyway.

Build the NVML sampler — a background thread that captures GPU state 10×/sec

A sampler pattern is the right shape for audit: non-intrusive background thread, captures state at a fixed cadence into a list, sampling overhead is ~20µs per call so it doesn't perturb the workload. Context manager boundaries (__enter__/__exit__) make it impossible to forget to stop the thread.

import pynvml, threading, time, torch, torch.nn as nn

pynvml.nvmlInit()
handle = pynvml.nvmlDeviceGetHandleByIndex(0)

def sample_once():
    util = pynvml.nvmlDeviceGetUtilizationRates(handle)
    mem = pynvml.nvmlDeviceGetMemoryInfo(handle)
    temp = pynvml.nvmlDeviceGetTemperature(handle, pynvml.NVML_TEMPERATURE_GPU)
    power_w = pynvml.nvmlDeviceGetPowerUsage(handle) / 1000.0
    return {
        'timestamp': time.monotonic(),
        'gpu_util': int(util.gpu),
        'mem_util': int(util.memory),
        'memory_pct': 100.0 * mem.used / mem.total,
        'temperature_c': int(temp),
        'power_w': float(power_w),
    }

class Sampler:
    def __init__(self, hz=10):
        self.samples = []
        self.stop_flag = threading.Event()
        self.thread = threading.Thread(target=self._run, args=(1.0 / hz,), daemon=True)
    def _run(self, period):
        while not self.stop_flag.is_set():
            self.samples.append(sample_once())
            time.sleep(period)
    def __enter__(self):
        self.thread.start(); return self
    def __exit__(self, *a):
        self.stop_flag.set(); self.thread.join(timeout=2)

The deliberately-bad training loop

A 40ms sleep before each GPU step simulates slow CPU-side preprocessing (tokenization, augmentation). That's the #1 reason production training GPUs idle. We run it for 15s while the sampler captures utilization so we can see the dips.

# Deliberately-bad training loop: dataloader slower than the GPU can consume.
device = 'cuda'
model = nn.Sequential(nn.Linear(2048, 4096), nn.GELU(), nn.Linear(4096, 1000)).to(device)
opt = torch.optim.SGD(model.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()

def bad_training_step():
    # Simulate slow CPU-side preprocessing (tokenization, augmentation, etc.)
    time.sleep(0.04)
    x = torch.randn(32, 2048, device=device)
    y = torch.randint(0, 1000, (32,), device=device)
    opt.zero_grad()
    loss = loss_fn(model(x), y)
    loss.backward()
    opt.step()
    torch.cuda.synchronize()

# Warm up so first-batch CUDA init isn't counted
for _ in range(3):
    bad_training_step()

with Sampler(hz=10) as s:
    t0 = time.time()
    while time.time() - t0 < 15:
        bad_training_step()

samples = s.samples
utils = [r['gpu_util'] for r in samples]
print(f"Collected {len(samples)} samples over {samples[-1]['timestamp']-samples[0]['timestamp']:.1f}s")
print(f"gpu_util: min={min(utils)}% max={max(utils)}% mean={sum(utils)/len(utils):.1f}%")