GPU Acceleration for NCP-GENL: Distributed Training & Parallelism Strategies

GPU 0: Full Model Copy  ←→  Data Batch 0  →  Gradients 0  ╮
GPU 1: Full Model Copy  ←→  Data Batch 1  →  Gradients 1  ├→  AllReduce  →  Update
GPU 2: Full Model Copy  ←→  Data Batch 2  →  Gradients 2  ╯

Layer N Weight Matrix [4096 x 4096]
├── GPU 0: [4096 x 2048]  (left half)
└── GPU 1: [4096 x 2048]  (right half)
    → AllReduce partial results → Output

Time →
GPU 0: [Micro-batch 0][Micro-batch 1][Micro-batch 2][   idle   ]
GPU 1: [   idle   ][Micro-batch 0][Micro-batch 1][Micro-batch 2]
GPU 2: [   idle   ][   idle   ][Micro-batch 0][Micro-batch 1]

4. Zero Redundancy Optimizer (ZeRO) — DeepSpeed

ZeRO is not technically a parallelism strategy but an optimizer memory optimization that eliminates redundancy in data parallelism. It is heavily tested on the NCP-GENL exam because it dramatically extends the scalability of data parallelism.

ZeRO Stages:

ZeRO-1: Each GPU stores only 1/N of the optimizer states (momentum and variance). Parameters and gradients remain fully replicated. Reduces memory by ~4x for Adam optimizer.

ZeRO-2: Additionally partitions gradients. Each GPU only stores gradients for its assigned parameter partition. Reduces memory by ~8x.

ZeRO-3 (ZeRO-Infinity): Partitions everything — parameters, gradients, and optimizer states. Each GPU stores only 1/N of the total state. Can offload to CPU or NVMe for even larger models. Reduces memory by ~N (number of GPUs).

Combining Parallelism Strategies: 3D Parallelism

Production training of very large models (100B+) combines multiple parallelism strategies. This is called 3D parallelism and is a frequent exam topic.

The Standard 3D Parallelism Configuration

For a 175B model on 64 GPUs (8 DGX nodes, 8 GPUs each):

64 GPUs Total = TP:8 x PP:4 x DP:2

Within each node (8 GPUs, NVLink):
  └── Tensor Parallelism across 8 GPUs

Across nodes (InfiniBand):
  └── Pipeline Parallelism across 4 stages (4 nodes per pipeline)

Data Parallelism:
  └── 2 parallel pipeline replicas processing different data

Why this configuration:

• TP=8 within nodes: Tensor parallelism requires high bandwidth. NVLink within a DGX node provides 600-900 GB/s.
• PP=4 across nodes: Pipeline parallelism only sends activations between adjacent stages, requiring less bandwidth than tensor parallelism. InfiniBand (200-400 Gb/s) is sufficient.
• DP=2 for remaining GPUs: Data parallelism uses the remaining parallelism dimension to increase effective batch size.

Configuration formula: Total_GPUs = TP x PP x DP

DeepSpeed and Megatron-LM

DeepSpeed

DeepSpeed is Microsoft's distributed training library, built on top of PyTorch. The NCP-GENL exam tests the following DeepSpeed features:

ZeRO Optimizer (covered above): The most tested DeepSpeed feature. Know all three stages and their memory/communication trade-offs.

DeepSpeed-Inference: Optimized inference engine with tensor parallelism, INT8 quantization, and kernel injection for transformer models.

Mixture of Experts (MoE) Support: DeepSpeed provides efficient MoE training with expert parallelism, which distributes different experts across different GPUs.

Activation Checkpointing Integration: Trades compute for memory by recomputing activations during the backward pass instead of storing them.

Megatron-LM

Megatron-LM is NVIDIA's library for training large transformer models. It provides:

Efficient Tensor Parallelism: Megatron-LM implements column-parallel and row-parallel linear layers that split matrix multiplications across GPUs with minimal communication:

• Column-parallel: Splits the weight matrix along columns. Each GPU computes a portion of the output, then results are gathered.
• Row-parallel: Splits along rows. Each GPU holds partial inputs, computes locally, then all-reduces.

Sequence Parallelism: Extends tensor parallelism to the non-tensor-parallel regions of the transformer (LayerNorm, dropout) by splitting the sequence dimension. This eliminates redundant computation in these regions.

Interleaved Pipeline Schedule (1F1B): An improved pipeline schedule that interleaves forward and backward passes to reduce the pipeline bubble. Each GPU alternates between forward passes on new micro-batches and backward passes on completed ones.

Megatron-LM + DeepSpeed Integration: In practice, many large-scale training setups use Megatron-LM for tensor and pipeline parallelism combined with DeepSpeed ZeRO for optimizer state management. NVIDIA NeMo Framework provides this integration out of the box.

GPU Memory Optimization Techniques

Beyond parallelism, several techniques reduce GPU memory consumption during training. These are frequently tested alongside parallelism questions.

Gradient Checkpointing (Activation Recomputation)

Instead of storing all intermediate activations for the backward pass (which can consume 30-60% of GPU memory), gradient checkpointing stores only a subset and recomputes the rest during backward.

Trade-off: Reduces activation memory by 60-80% but increases training time by 20-35% due to recomputation.

When to use: When GPU memory is the bottleneck preventing a larger batch size or model. Nearly always enabled for models larger than 13B parameters.

Mixed-Precision Training

Mixed-precision training uses FP16 or BF16 for forward/backward computation while maintaining FP32 master weights and optimizer states.

BF16 vs FP16:

• BF16 (Brain Float 16): Same exponent range as FP32 (8 bits), reduced mantissa (7 bits vs 23). No loss scaling needed. Preferred on Ampere+ GPUs.
• FP16: Smaller exponent range (5 bits), higher mantissa precision (10 bits). Requires loss scaling to prevent gradient underflow.

Flash Attention

Flash Attention is a memory-efficient attention implementation that avoids materializing the full N x N attention matrix:

• Standard attention: O(N^2) memory for the attention matrix (N = sequence length)
• Flash Attention: O(N) memory by computing attention in tiles and fusing operations

For a sequence length of 8,192 tokens with hidden dimension 4,096, standard attention stores a 8192 x 8192 matrix per attention head (~256MB per head). Flash Attention reduces this by 10-100x depending on sequence length.

NVIDIA Nsight for GPU Profiling

The exam tests your ability to identify GPU performance bottlenecks using NVIDIA profiling tools.

Nsight Systems

Nsight Systems provides system-level profiling: CPU/GPU timeline, kernel launch latency, memory transfers, NCCL communication.

Key metrics to monitor:

• GPU utilization: Percentage of time GPU SMs are active. Target: >80%
• Tensor Core utilization: Percentage of compute using Tensor Cores. Target: >60%
• Memory bandwidth utilization: Percentage of theoretical bandwidth used. Bottleneck if >80%
• NCCL communication time: Time spent in inter-GPU communication. Problematic if >20% of step time

Nsight Compute

Nsight Compute provides kernel-level profiling: instruction throughput, memory access patterns, occupancy, warp efficiency.

Common bottlenecks the exam asks about:

NCCL Communication Optimization

NCCL (NVIDIA Collective Communications Library) handles inter-GPU communication for distributed training. The exam tests:

Communication Primitives:

• All-Reduce: Aggregate gradients across all GPUs (data parallelism). Volume = 2 x model_size.
• All-Gather: Collect partitioned parameters (ZeRO-3). Volume = model_size.
• Reduce-Scatter: Distribute reduced gradients (ZeRO-2). Volume = model_size.
• Point-to-Point: Send activations between pipeline stages. Volume = activation_size per stage.

Optimization Strategies:

• Communication-computation overlap: Start communicating gradients for completed layers while computing gradients for remaining layers
• Gradient bucketing: Group small gradient tensors into larger buckets for more efficient all-reduce
• Hierarchical all-reduce: Reduce within nodes (NVLink) first, then across nodes (InfiniBand)

DGX System Configuration

The exam assumes familiarity with NVIDIA DGX hardware:

Key hardware facts for the exam:

• 8 GPUs per DGX node connected by NVLink (high bandwidth, within node)
• Nodes connected by InfiniBand (lower bandwidth, between nodes)
• Tensor parallelism: within a node (NVLink)
• Pipeline parallelism: across nodes (InfiniBand)
• This hardware topology dictates the parallelism configuration

Practice Questions

Question 1: You are training a 70B parameter model on 32 A100 80GB GPUs (4 DGX nodes). Which parallelism configuration maximizes training throughput?

A) DP=32 (pure data parallelism) B) TP=8, PP=2, DP=2 C) TP=4, PP=8, DP=1 D) TP=32 (pure tensor parallelism)

Answer: B. Pure DP (A) is impossible because the 70B model with Adam optimizer requires ~1,120GB, far exceeding a single GPU's 80GB. Pure TP (D) across 32 GPUs spans multiple nodes, and tensor parallelism is inefficient across InfiniBand. TP=4, PP=8 (C) under-utilizes NVLink within each 8-GPU node. TP=8 within each node maximizes NVLink bandwidth, PP=2 splits across 2 pairs of nodes with manageable pipeline bubble, and DP=2 doubles the effective batch size.

Question 2: During distributed training of a 175B model using pipeline parallelism with 16 stages and 32 micro-batches, what is the pipeline efficiency?

A) 50% B) 68% C) 82% D) 94%

Answer: B. Pipeline efficiency = m / (m + p - 1) = 32 / (32 + 16 - 1) = 32 / 47 = 68%. To improve this, increase micro-batches (e.g., m=64 gives 64/79 = 81%) or reduce pipeline stages.

Question 3: A training run on 8 A100 GPUs shows 40% GPU utilization and 60% of step time spent in NCCL all-reduce. What is the most effective optimization?

A) Switch from data parallelism to pipeline parallelism B) Enable gradient bucketing and overlap communication with backward pass computation C) Increase the batch size per GPU D) Switch from BF16 to FP32 precision

Answer: B. The bottleneck is NCCL communication (60% of step time). Gradient bucketing reduces the overhead of many small all-reduce operations, and overlapping communication with computation hides latency by starting gradient synchronization for early layers while computing gradients for later layers. Option A does not address the communication overhead directly. Option C would increase computation time, improving the compute-to-communication ratio, but B is more targeted. Option D would slow training further.

For more practice questions covering all 10 NCP-GENL domains, try our NCP-GENL practice exams.

Summary: GPU Acceleration Key Takeaways

For the complete preparation strategy, see our How to Pass NCP-GENL guide and 8-Week NCP-GENL Study Plan.