NVIDIA-Certified Associate: Generative AI with LLMs

It initializes all weights to zero across every layer to ensure perfectly symmetric learning signals across neurons, guaranteeing that each unit receives identical gradient updates during early training phases

It sets all weights to small constant values like 0.01 across the entire network to prevent gradient explosion, ensuring that activations remain bounded throughout the forward propagation process

It sets weights proportional to the number of input and output connections, helping maintain consistent variance of activations and gradients across layers

It initializes weights randomly from a standard uniform distribution between -1 and 1 regardless of network dimensions, providing sufficient randomness to break symmetry between neurons in each layer