Step 1: Raw GPU telemetry via NVML
You cannot optimize what you cannot measure. The most common AI-ops failure mode isn't a crash — it's a training run that appears fine but sits at 30% GPU utilization because the CPU-side data loader is the real bottleneck. At cloud GPU rates ($1–3/hr for an A100), a 30%-utilized cluster is a 70% tax on every project.
The production-grade answer is DCGM + Prometheus + Grafana — NVIDIA's Data Center GPU Manager exports per-GPU telemetry to a time-series database, and Grafana renders dashboards + alerts. In this lab you'll build up the same stack from scratch in Python so you understand each layer, then see the production docker-compose topology.
Why this lab is the foundation of the Infra track
Every downstream infrastructure lab uses GPU telemetry:
- Profiling (Nsight) — needs util/mem traces to interpret
- DALI tuning — measures itself by moving the utilization line
- Cost audit — cost/token = $-per-GPU-hour ÷ (throughput × utilization)
- Health checks — "is this GPU dying?" is a metrics question
- Auto-scaling — HPA decisions key off metrics
References
- NVIDIA DCGM (Data Center GPU Manager) — the official C/Python API for GPU telemetry, health, policy. What every production deployment uses.
- dcgm-exporter on GitHub — the Prometheus exporter that wraps DCGM. Deploy as a DaemonSet in Kubernetes; ~50 metrics per GPU.
- NVML Python bindings (pynvml) — what we'll use directly in this lab. Same API as
nvidia-smiuses. - Prometheus exposition format — the plain-text spec that lets any language export metrics.
- Karpathy — Bandwidth Is All You Need — not strictly about metrics, but makes the case why GPU utilization is the single number that matters most in training.
What you'll build
- Raw GPU telemetry — pull utilization, memory, power, temperature via NVML
- Time-series sampler — poll every 500 ms during a real workload; plot the trace
- The classic bottleneck story — show a slow-dataloader training loop (sawtooth utilization) vs an optimized one (steady high utilization)
- Production bridge — expose the same metrics in Prometheus format over HTTP; scrape with
curl
NVML (NVIDIA Management Library) is the C API under nvidia-smi. pynvml is its Python wrapper. Everything DCGM and dcgm-exporter do, you can also do with ~20 lines of NVML calls.
Key metrics and what they mean:
| Metric | NVML call | Interpretation |
|---|---|---|
| Utilization (SM) | nvmlDeviceGetUtilizationRates | % of time SMs were doing work. <70% = investigate |
| Memory used | nvmlDeviceGetMemoryInfo | How much VRAM your process holds |
| Power draw | nvmlDeviceGetPowerUsage | Watts. Full tensor cores draw near TDP |
| Temperature | nvmlDeviceGetTemperature | Throttles near ~85C on consumer cards |
| SM clock | nvmlDeviceGetClockInfo | Throttles if temperature or power hit limits |
| Encoder/decoder util | nvmlDeviceGetEncoderUtilization | Only non-zero for video workloads |
Don't miss: utilization is time-based, not throughput-based. A GPU can report 100% util while running at 10% of peak FLOPS (e.g., memory-bound kernels). Always pair with workload-specific throughput metrics (tokens/sec, images/sec).
import pynvml
pynvml.nvmlInit()
handle = pynvml.nvmlDeviceGetHandleByIndex(0)
def gpu_snapshot():
name = pynvml.nvmlDeviceGetName(handle)
# pynvml returns bytes on some versions, str on others
if isinstance(name, bytes):
name = name.decode()
mem = pynvml.nvmlDeviceGetMemoryInfo(handle)
util = pynvml.nvmlDeviceGetUtilizationRates(handle)
power_mw = pynvml.nvmlDeviceGetPowerUsage(handle)
temp = pynvml.nvmlDeviceGetTemperature(handle, pynvml.NVML_TEMPERATURE_GPU)
clock_sm = pynvml.nvmlDeviceGetClockInfo(handle, pynvml.NVML_CLOCK_SM)
return {
'name': name,
'memory_total_mib': mem.total // (1024 * 1024),
'memory_used_mib': mem.used // (1024 * 1024),
'memory_free_mib': mem.free // (1024 * 1024),
'utilization_gpu_pct': util.gpu,
'utilization_memory_pct': util.memory,
'power_draw_w': power_mw / 1000.0,
'temperature_c': temp,
'sm_clock_mhz': clock_sm,
}
gpu_info = gpu_snapshot()
for k, v in gpu_info.items():
print(f' {k:25s} {v}')
