PreporatoPreporato
NVIDIA DALI: GPU-Accelerated Data Pipelines
GPU sandbox · jupyter
Beta

NVIDIA DALI: GPU-Accelerated Data Pipelines

Move image decoding, resizing, and augmentation from CPU to GPU with NVIDIA DALI, and benchmark it against a standard PyTorch DataLoader. The input-pipeline fix that unlocks real multi-GPU throughput.

30 min·4 steps·2 domains·Intermediate·ncp-genlncp-adsnca-genl

What you'll learn

  1. 1
    Baseline: PyTorch DataLoader + torchvision transforms
  2. 2
    Build a DALI pipeline (GPU decoding + augmentation)
  3. 3
    Benchmark DALI against the PyTorch baseline
  4. 4
    When is DALI worth the engineering cost?

Prerequisites

  • Comfortable with PyTorch DataLoader and torchvision transforms
  • Basic understanding of JPEG decoding and image augmentation
  • CUDA-capable GPU with a working PyTorch installation

Exam domains covered

GPU Acceleration & Distributed TrainingGPU Infrastructure & Operations

Skills & technologies you'll practice

This intermediate-level gpu lab gives you real-world reps across:

DALInvJPEGData PipelineThroughputPyTorchAugmentationBenchmarking

What you'll build in this NVIDIA DALI lab

Input-pipeline starvation is the silent killer of GPU utilization — the moment you scale to 4+ GPUs, the same CPU-side JPEG decode that looked fine on one card collapses into 40-70% idle SMs while your DataLoader workers max out CPU cores. DALI is the fix engineers reach for when nvidia-smi says the GPU is bored and the CPU is on fire. You'll walk away with a working GPU-native preprocessing pipeline built on nvJPEG + CUDA augmentations, an honest before/after throughput delta (typically 1.3-3× on a single card, 3-10× once multiple GPUs share one host), a DALIGenericIterator you can drop into an existing PyTorch training loop, and — critically — the judgment to know when DALI is overkill versus a 10× win. About 30 minutes on a live NVIDIA GPU pod we hand you; PyTorch, DALI, and nvJPEG are already installed.

The substance is the pipeline wiring: @pipeline_def(batch_size=32, num_threads=4, device_id=0) with fn.readers.file → fn.decoders.image(device='mixed') → fn.resize → fn.random_resized_crop → fn.crop_mirror_normalize(output_layout='CHW'). device='mixed' is the magic word — it routes JPEG decode to nvJPEG's dedicated hardware decoders and keeps every downstream op on the GPU, so there's no H2D copy of raw pixels and no CPU-side PIL bottleneck. The trap most teams hit: DALI pays pipeline-prefill cost on the first batch (CUDA kernel warmup, async prefetcher kickoff), so if you skip warmup your benchmark makes DALI look slower than torchvision. The other trap: DALI only helps data-pipeline-bound workloads — vision training on A100s wins big, but BERT fine-tuning (tokenization is cheap, there's no preprocessing) sees zero benefit. You'll measure both and build a mental model for where the advantage actually transfers.

Prereqs: PyTorch DataLoader and torchvision transforms, a rough sense of JPEG decoding and augmentation pipelines, a CUDA-capable GPU. Preinstalled: PyTorch, NVIDIA DALI, nvJPEG bindings, JupyterLab. The grader enforces real behavior rather than code-shape matching — the dataset must have ≥100 JPEGs under dataset_dir, the first DALI batch must arrive on CUDA as a 4D tensor with batch size ≥16, dali_throughput must exceed pytorch_throughput by at least 1.1×, and you have to produce substantive where_dali_wins / where_dali_loses lists that show you understand the modality dependency.

Frequently asked questions

Is DALI worth it if I'm not training on ImageNet?

Depends on whether your pipeline is CPU-bound. If nvidia-smi shows sustained sub-70% GPU utilization while your DataLoader workers are pegged, you're starved and DALI helps. If you're fine-tuning BERT where the 'preprocessing' is tokenization — almost no CPU work — DALI gives you nothing. The Step 4 reflection asks for this analysis explicitly: define the acceptance metric (images/sec at fixed batch size, or wall-clock epoch time) before you integrate, so you can tell afterward whether it paid off.

What's nvJPEG and why does it matter?

nvJPEG is NVIDIA's GPU-accelerated JPEG decoder — it reads compressed bytes and produces decoded RGB tensors directly in GPU memory, bypassing libjpeg-turbo on the CPU. Dedicated hardware decoders on newer GPUs make this dramatically faster than a CPU decode + H2D copy, and it's what fn.decoders.image with device='mixed' triggers in a DALI pipeline. The Step 2 pipeline uses it implicitly; you're not wiring it by hand, but it's the piece doing the heavy lifting.

Why does the PyTorch DataLoader look slow in this lab?

Because the Step 1 baseline intentionally uses the simplest possible configuration — one worker, synchronous CPU transforms — so the comparison is clean. A fully-tuned PyTorch DataLoader with num_workers=8, pin_memory=True, non_blocking=True, prefetch_factor=2 closes most of the gap for many workloads. The point of the lab isn't 'DALI > DataLoader' unconditionally, it's measuring the delta on a controlled workload and then thinking carefully about where the advantage transfers.

What's the warmup step about in Step 3?

DALI pays pipeline-prefill latency on the very first batch — building the execution graph, warming the CUDA kernels for each op, kicking off the async prefetcher. If you time starting from iteration zero, that prefill amortizes badly over a tiny test set and DALI looks slower than it is. One full warmup iteration, then time the real epochs — exactly the same discipline you'd apply to any async GPU benchmark, including the PyTorch baseline.

Can I mix DALI with my existing PyTorch training loop?

Yes, and that's the whole point of DALIGenericIterator (or DALIClassificationIterator for classification, or the PyTorch Lightning adapters). It yields dicts/lists of CUDA tensors that drop directly into your existing forward pass — you replace the DataLoader, nothing else. For multi-GPU you'd use DALIGenericIterator with num_shards/shard_id matching your DDP rank so each process reads its own shard of the dataset.

What does the grader actually check?

Step 1 walks dataset_dir recursively for ≥100 JPEG/JPEG files and verifies pytorch_throughput > 0. Step 2 confirms the DALI iterator was built and first_batch is a 4D CUDA tensor with batch size ≥16. Step 3 enforces dali_throughput > pytorch_throughput * 1.1 — a modest floor that makes sure both pipelines ran on the same data. Step 4 requires where_dali_wins with ≥3 scenarios and where_dali_loses with ≥2 anti-cases as strings.