Four new nodes and no speed-up

Wrenfield Robotics (fictional) trains a 20-billion-parameter vision-language-action model for warehouse robots. Its cluster runs Kubernetes with the NVIDIA GPU Operator and Network Operator. Each node has 8 H100 SXM GPUs and 8 RoCE network cards at 400 Gb/s, one per GPU, on a rail-optimized Ethernet fabric. Three weeks ago they added a fourth rack: nodes 13 to 16.

Assume:

• Per GPU: 989 TFLOPS of dense BF16. Plan on 40% MFU.
• The job uses fully sharded data parallel (FSDP) across all GPUs, with BF16 weights and gradients. Each step runs three collectives of 40 GB each: an all-gather of the weights in the forward pass, another in the backward pass, and a reduce-scatter of the gradients.
• For this model, communication does not overlap with compute. Step time is about compute plus communication.
• The global batch is 512 sequences of 2,048 tokens on any number of nodes.
• A collective of S bytes across n GPUs takes about S × (n − 1) / n divided by the bus bandwidth that nccl-tests reports for all-reduce. Assume the all-gather and the reduce-scatter reach the same bus bandwidth.

What the customer tells you:

• Last month, on nodes 1 to 12 (96 GPUs): 3.65 s per step.
• This week, on nodes 1 to 16 (128 GPUs): 3.55 s per step. "We added a third more GPUs and got 3%."
• nccl-tests all-reduce at 8 GB on all 16 nodes: 112 GB/s of bus bandwidth. The acceptance report for nodes 1 to 12 said 352 GB/s.
• Their head of infrastructure says the new rack's switch ports are misconfigured.

The interviewer answers your questions and gives you the result of any check you ask for.

Asks:

1. Before any check: what step time do you expect on 16 nodes, and what does the gap tell you?
2. Walk through your diagnosis: what you check, with which tool, and why, in order.
3. Name the root cause, the fix, and how you prove the fix worked.
4. What do you put in place so this cannot happen again?

Write down the step time you expect on 16 nodes, and the first three checks you would ask for, in order, with the tool for each.