NVIDIA DALI: GPU-Accelerated Data Pipelines

In Lab I-1 you saw the data-loader bottleneck: GPU at 30% utilization while the CPU struggles to decode JPEGs and apply augmentations. The classic fix is num_workers=8 on PyTorch's DataLoader, which helps — but you're still pinning 8 CPU cores to a job the GPU could do faster.

NVIDIA DALI (Data Loading Library) moves decoding, resizing, color-space conversion, cropping, and normalization onto the GPU itself via nvJPEG and custom CUDA kernels. Once DALI is in the pipeline, the CPU just reads bytes off disk; everything else happens on-device. The practical effect on real ImageNet-scale training:

That's a 4–6× throughput multiplier on data-pipeline-bound workloads. For any serious multi-GPU training setup, DALI is not optional.

When DALI is worth it (and when it isn't)

• Wins big: Vision workloads on multi-GPU, large images (ImageNet/COCO/video), aggressive augmentation
• Marginal gain: Small images (CIFAR-sized), tiny models, dev-machine single-GPU experiments
• Overkill: NLP workloads (tokenization is fast; CPU is plenty). Reinforcement learning (env step, not data pipe, is the bottleneck)

This lab runs on a single 8 GB 3060 Ti, so we'll see a modest speedup — enough to demonstrate the pattern. On an A100 with 8× the compute, the relative gap grows dramatically.

References

• NVIDIA DALI docs — official reference.
• DALI examples repo — production ImageNet + COCO pipelines.
• nvJPEG — the hardware-accelerated JPEG decoder DALI uses. A100/H100 have dedicated JPEG decode engines.
• PyTorch DataLoader tuning guide — before you reach for DALI, squeeze num_workers + pin_memory + persistent_workers first.
• Horace He — Making Deep Learning Go Brrrr — the mental model: input-pipe throughput caps training throughput.

What you'll build

1. Synthesize a ~1,000-JPEG dataset; measure PyTorch DataLoader baseline throughput
2. Build a DALI pipeline that reads from disk and processes on GPU
3. Benchmark DALI; compute the speedup
4. Reason about when DALI is worth the engineering investment

---

We'll generate a tiny synthetic JPEG dataset (512 images, 224×224) and use a standard PyTorch DataLoader with torchvision transforms. This is the canonical 'ship it' pipeline in ~90% of PyTorch tutorials.

import os, shutil, time
import numpy as np
from PIL import Image

# Synth dataset: random RGB noise saved as JPEGs. Mimics an image-classification input.
dataset_dir = '/tmp/dali_dataset'
if os.path.exists(dataset_dir):
    shutil.rmtree(dataset_dir)
os.makedirs(dataset_dir + '/class0', exist_ok=True)

N_IMAGES = 512
IMG_SIZE = 224
for i in range(N_IMAGES):
    # Random noise image
    arr = np.random.randint(0, 255, (IMG_SIZE, IMG_SIZE, 3), dtype=np.uint8)
    Image.fromarray(arr).save(f'{dataset_dir}/class0/{i:04d}.jpg', quality=85)

total_bytes = sum(os.path.getsize(f'{dataset_dir}/class0/{f}') for f in os.listdir(dataset_dir + '/class0'))
print(f'Created {N_IMAGES} JPEGs in {dataset_dir} ({total_bytes/1024/1024:.1f} MB total)')

The CPU-bound baseline — torchvision + DataLoader

The standard PyTorch data pipeline: a Dataset that reads files one at a time and applies transforms.Compose(...) entirely on the CPU. Works everywhere, but every op is CPU: JPEG decode, resize, random crop, flip, normalize. On a fast GPU the CPU becomes the bottleneck — you'll see GPU util dip to 20-40% while the dataloader catches up.

# Baseline: PyTorch Dataset/DataLoader with torchvision transforms (all CPU)
import torch
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms

class JPEGDataset(Dataset):
    def __init__(self, root, transform):
        self.files = sorted(os.path.join(root, 'class0', f) for f in os.listdir(os.path.join(root, 'class0')) if f.endswith('.jpg'))
        self.transform = transform
    def __len__(self): return len(self.files)
    def __getitem__(self, i):
        img = Image.open(self.files[i]).convert('RGB')
        return self.transform(img)

# Realistic production-style CPU transforms
tx = transforms.Compose([
    transforms.Resize(256),
    transforms.RandomCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

ds = JPEGDataset(dataset_dir, tx)
loader = DataLoader(ds, batch_size=32, num_workers=0, pin_memory=True)

Measure — 2 full epochs

Warm the pipeline once (first iter pays worker spawn + pipeline fill overhead), then time 2 full epochs. torch.cuda.synchronize() after each batch makes the wall-clock measurement reflect real end-to-end throughput.

# Warm up (let workers spawn + pipeline fill)
for b in loader:
    b.cuda(non_blocking=True)
    break

# Benchmark: time 2 full epochs
n_images = 0
t0 = time.perf_counter()
for _ in range(2):
    for b in loader:
        b = b.cuda(non_blocking=True)
        torch.cuda.synchronize()
        n_images += b.shape[0]
pytorch_throughput = n_images / (time.perf_counter() - t0)
print(f'PyTorch DataLoader + torchvision (CPU): {pytorch_throughput:.1f} images/sec')