Step 1: Instrument a training workload with NVTX ranges
Metrics (Lab I-1) tell you something is slow. Profiling tells you exactly which nanosecond is slow. NVIDIA Nsight Systems (nsys) is the timeline-level profiler that's standard in every serious CUDA shop — it records every CUDA API call, every kernel launch, every memory copy, every NVTX range, and renders them on a unified timeline you can zoom into.
This lab teaches the profile-then-optimize workflow on a deliberately suboptimal training loop. You'll find the bottleneck (GPU sits idle waiting for the CPU dataloader), apply a one-line fix, and measure the throughput jump.
Why Nsight Systems and not just time.perf_counter()?
Timing calls measure wall-clock time across the whole step. Nsight measures:
- CUDA kernel execution time per kernel, aggregated (you can see which kernel eats the most GPU time)
- Launch gaps — time between the end of one kernel and the start of the next, which is either CPU overhead or a sync barrier
- Memory transfers (H2D/D2H) overlapped or not with compute
- CPU thread activity — are Python worker threads blocked on the GIL?
- NVTX ranges you annotate your code with (
torch.cuda.nvtx.range_push/pop) so you see logical phases (forward, backward, optimizer)
Two modes we'll use:
nsys profile --stats=true ...— produces a.nsys-repreport + prints summary stats to stdout. What you'll use 95% of the time.- Nsight Systems GUI — open the
.nsys-repin the desktop app for the interactive timeline. Not demonstrated here (no display), but mention in the references.
References
- Nsight Systems User Guide — the complete reference.
- Karpathy — Recipe for Training Neural Networks — not Nsight-specific but makes the case that profiling before optimizing is non-negotiable.
- NVTX ranges in PyTorch — how we'll annotate phases.
- "GPU Performance Background" — NVIDIA deep-learning perf guide — the mental model you need before reading a profile.
Without NVTX, an Nsight profile is a sea of cudaLaunchKernel events. With NVTX ranges, you see your code's logical phases ("data_load", "forward", "backward", "optimizer") as colored bars on the timeline. Every serious CUDA project ships with NVTX.
We'll write a small training script with NVTX annotations, save it to a file, and verify nsys is on the PATH.
import subprocess, os
# Confirm nsys is available
nsys_result = subprocess.run(['nsys', '--version'], capture_output=True, text=True)
nsys_version = nsys_result.stdout.strip()
print(f'nsys version: {nsys_version}')
# The workload we'll profile lives at workspace/scripts/bad_training.py. It's a
# deliberately-bad training loop:
# - CPU-side preprocessing that blocks the main thread each iteration
# - Slow host-to-device transfers (non-pinned memory)
# - A plain `for`-loop dataloader that does all work synchronously
workload_script_path = '/home/labuser/workspace/scripts/bad_training.py'
assert os.path.exists(workload_script_path), f'workload script not found at {workload_script_path}'
print(f'Workload script: {workload_script_path} ({os.path.getsize(workload_script_path)} bytes)')
from IPython.display import Code
Code(filename='/home/labuser/workspace/scripts/bad_training.py', language='python')
