GPUs at 33% on a satellite imagery job

Orbel Earth Imaging (fictional) trains a segmentation model on satellite images to map flooded fields. Training runs on Kubernetes: 4 nodes, each with 8 H100 GPUs and 96 CPU cores. Each node runs one pod with 8 GPUs and 8 training processes (PyTorch DistributedDataParallel, one process per GPU).

What the customer tells you:

• Data: image chips of 512 × 512 pixels with 13 spectral bands of 16-bit values, stored as compressed files of about 3.4 MB each on an NFS server that can deliver 12 GB/s.
• Batch: 8 chips per GPU per step, 256 per step in all.
• A benchmark with synthetic data already in GPU memory: 0.62 s per step.
• Real training: 1.9 s per step. GPU utilization in nvidia-smi swings between 0 and 100% and averages 33%.
• Data loading: a PyTorch DataLoader with 4 workers per training process. Each chip is decompressed and augmented on the CPU (random rotation, flips, normalization per band in float32).
• The ML lead says: "Storage is too slow. We want to buy an all-flash file system."

The interviewer answers your questions and gives you the result of any check you ask for.

Asks:

1. Before any check: how much data per second does training need at the target speed? Is the NFS server a plausible bottleneck?
2. Walk through your diagnosis, with the tool for each step.
3. Name the root cause and the fix, with the step time you expect after the fix.
4. What do you tell the ML lead about the all-flash purchase?

Write down the read bandwidth training needs at the target speed, and the CPU cores the data pipeline needs per node. Then decide whether the customer's storage theory survives that math.