Step 1: Anatomy of the GPU Operator chain
Anatomy of the GPU Operator chain
The NVIDIA GPU Operator is a single Helm install that deploys a half-dozen cooperating components. Once installed, you don't interact with it directly — you interact with the evidence it leaves on your cluster: node labels, Capacity values, a RuntimeClass, Prometheus metrics endpoints. As a platform engineer, your job is to read that evidence and trace each piece back to the component that produced it. That's the muscle memory this lab builds.
What the Operator deploys (and what each part contributes)
| Component | Lives where | Contributes |
|---|---|---|
| NVIDIA driver (host-level or daemonset) | Each GPU node | Kernel module + libcuda. Without it, no /dev/nvidia* device files exist. |
| NVIDIA Container Toolkit | Each GPU node | Modifies OCI container specs at start time to inject GPU device files and library mounts. |
| Device plugin | DaemonSet on GPU nodes | Tells the kubelet "this node has N GPUs," which becomes Capacity: nvidia.com/gpu: N on the Node object. |
| gpu-feature-discovery (GFD) | DaemonSet on GPU nodes | Writes node labels: nvidia.com/gpu.product, nvidia.com/gpu.memory, nvidia.com/gpu.count, nvidia.com/gpu.deploy.*. Lets workloads target specific GPU types. |
| dcgm-exporter | DaemonSet on GPU nodes | Exposes a /metrics endpoint with utilization, memory, temperature, ECC errors. Hooked by Prometheus. |
RuntimeClass nvidia | Cluster-wide | Cluster-scoped object that maps the nvidia runtime name to the kubelet's nvidia runtime handler (which is the NVIDIA Container Runtime). |
| Validator pod | Runs once at install | Sanity-checks that all of the above wired up correctly. Run kubectl get pods -n gpu-operator -l app=nvidia-operator-validator to see the result. |
| MIG manager (optional) | DaemonSet on MIG-capable nodes | Reconfigures GPUs into MIG instances based on a node label. Only matters on A100/H100/Hopper. |
The chain you care about for every GPU workload:
gpu-feature-discovery → node labels → your pod's nodeSelector
device plugin → Capacity + Allocatable → scheduler arithmetic
RuntimeClass nvidia → kubelet handler → NVIDIA Container Runtime → /dev/nvidia*
Three independent contributions, three independent failure modes. We'll read each one off the cluster in this step.
Five nodes, one real
kubectl get nodes lists five nodes. Four of them, the kwok-* ones, are simulated: they carry A100 and L40S labels for the multi-GPU targeting lab, but every one of them has a kwok.x-k8s.io/node=fake:NoSchedule taint, so nothing you deploy here lands on them. The fifth node is the real one. Everything in this lab runs there, and it is the node whose labels you have to read. The simulated nodes all share the type=kwok label, so kubectl get nodes -l 'type!=kwok' gives you the real node whenever you need it.
Read the cluster
Three commands give you the three pieces of evidence. Run each one in the terminal (or click Run to execute the whole batch at once):
# 1. Every node with its GPU model, GPU count and taint. Note which one has no taint.
kubectl get nodes -o custom-columns='NAME:.metadata.name,PRODUCT:.metadata.labels.nvidia\.com/gpu\.product,GPUS:.status.allocatable.nvidia\.com/gpu,TAINT:.spec.taints[*].key'
# 2. Everything the GPU Operator wrote on the real node: the `nvidia.com/gpu.*`
# labels come from gpu-feature-discovery, the `*.deploy.*` labels are the
# Operator's own scheduling toggles, and the `nvidia.com/gpu` rows under
# Capacity and Allocatable come from the device plugin.
kubectl describe $(kubectl get nodes -l 'type!=kwok' -o name) | grep nvidia.com
# 3. RuntimeClasses — cluster-scoped runtime selectors. The `nvidia` row is
# what GPU pods reference; the `handler` column is the runtime *name* the
# kubelet looks up in its own config.
kubectl get runtimeclass
After running, take a moment to think about what would happen to a GPU pod if each of these three (labels, Capacity, RuntimeClass) were missing. Use the table above to map your prediction. The quiz below pins the cluster's actual values into your head.
Why this matters in practice: when a GPU workload won't run, the chain has three fundamentally different breakage points. Knowing which evidence to inspect first (
describe nodefor labels and Capacity,get runtimeclassfor the runtime path) cuts diagnosis from minutes to seconds. The NCA-AIIO exam tests this through scenarios where you're given a symptom ("Pod is Pending — Insufficient nvidia.com/gpu" vs "Running but no GPU mounted") and asked which component is at fault.
