TL;DR: The GPU Operator is a chain, and each link fails with a different symptom. No node labels means feature discovery. No nvidia.com/gpu under allocatable means the device plugin. A pod stuck Pending with "Insufficient nvidia.com/gpu" means the resource exists but is fully claimed, or was never advertised. A pod that reaches Running with no nvidia-smi inside it means the RuntimeClass, and it is the one that confuses people most because scheduling clearly worked. Start with kubectl describe node on a GPU node, because labels and allocatable together split the chain in half in one command.
The most disorienting failure in GPU Kubernetes is the one where everything looks fine. The pod is Running. The events are clean. The node has four A100s in it. And inside the container, nvidia-smi reports command not found.
That symptom has a specific cause, and finding it quickly depends on holding one idea: the operator is not a thing that either works or does not. It is a chain of cooperating components, each contributing a different piece of evidence, and a broken link produces a symptom that points at the link rather than at the pod complaining about it.
Work this on a real cluster
The AI Infrastructure Engineer course has you install the operator by hand, then break each link deliberately and identify it from the symptom alone. That exercise is worth more than any amount of reading, because the symptoms are only memorable once you have caused them.
The chain, in the order it runs
Four components matter for diagnosis. Each one leaves evidence you can read.
gpu-feature-discovery inspects the hardware and publishes node labels: nvidia.com/gpu.present, nvidia.com/gpu.product, nvidia.com/gpu.count, nvidia.com/gpu.memory and more. Those labels are a scheduling API rather than decoration, because they are what a manifest targets when it needs an A100 rather than any GPU.
The device plugin advertises the schedulable resource. It is what puts nvidia.com/gpu under the node's allocatable list, which is the only thing the scheduler can actually satisfy a request against.
RuntimeClass selects which container runtime handler runs the pod. Choosing the NVIDIA handler is what causes driver libraries and device nodes to be injected into the container. This step happens after scheduling and is entirely independent of it.
dcgm-exporter publishes telemetry. It has no role in whether a workload runs, which is worth knowing so you stop suspecting it during an outage.
The order matters because a failure early in the chain masks everything downstream. If nothing is advertised, no pod ever reaches a runtime, so a RuntimeClass problem cannot be observed until the device plugin problem is fixed.
The one command to run first
kubectl describe node <a-gpu-node>
Read two things from it. The labels block tells you whether feature discovery ran. The allocatable block tells you whether the device plugin ran. Those two facts split the chain in half, and you have spent one command to do it.
If both are present, the problem is downstream of scheduling and you are looking at the runtime path or the workload itself. If either is missing, you have found your link and nothing further along is worth investigating yet.
Symptom: the node has no GPU labels at all
What you see. kubectl describe node shows no nvidia.com/* labels. The node otherwise looks healthy.
What it means. Feature discovery is not running, or it is running and finding nothing. The second case is worth ruling out first, because it is not a Kubernetes problem: if the driver is not loaded on the host, there is no hardware for the component to describe. Check nvidia-smi on the node itself before touching anything in the cluster.
Why it matters beyond the labels. Placement rules break silently. A manifest with a nodeSelector on nvidia.com/gpu.product will sit Pending forever with a message about node affinity, and the cause is two components upstream of the pod you are looking at.
Symptom: no nvidia.com/gpu under allocatable
What you see. Labels are present, so the hardware is visible, but the allocatable block lists no nvidia.com/gpu.
What it means. The device plugin is not advertising. Common causes are the plugin pod not running on that node, the plugin running but unable to reach the driver, or a MIG configuration mismatch where the plugin is advertising a different resource name than you expect.
The MIG variant is worth calling out. On a MIG-enabled card the plugin's strategy decides what appears. Under the single strategy the node advertises nvidia.com/gpu. Under mixed it advertises per-profile resources such as nvidia.com/mig-1g.10gb instead. A cluster where somebody changed the strategy and nobody updated the manifests produces workloads that request a resource nothing publishes, which reads as a capacity problem and is a naming problem.
That failure has a quieter cousin: a card partitioned into seven instances while the plugin runs single strategy advertises one schedulable unit, so six instances sit idle while the scheduler reports the node full. Utilisation telemetry is what catches it, since the scheduling layer looks entirely healthy.
Symptom: pod Pending with "Insufficient nvidia.com/gpu"
What you see. The pod exists, kubectl get pods lists it, and the scheduling condition reports insufficient GPU across all nodes.
What it means. The scheduler understands what you asked for and cannot find it. That splits three ways.
The resource is genuinely all claimed, which is a capacity and priority question rather than an operator one. The resource is advertised on nodes your pod cannot reach, because of a taint you do not tolerate or a selector that excludes them. Or the resource is not advertised anywhere, which takes you back to the device plugin symptom above.
A useful distinction. Read whether the message mentions preemption. A line saying no preemption victims were found tells you the scheduler considered evicting something and decided nothing was lower priority than your pod, which points at priority configuration rather than at raw capacity. That is a different fix: your workload may be inheriting a low default priority class it never asked for.
Symptom: pod Running, but no GPU inside the container
What you see. The pod reaches Running. Events are clean. Inside the container, nvidia-smi is missing, or /dev/nvidia* device nodes do not exist, or a CUDA call reports no devices.
What it means. Scheduling succeeded, so the resource existed and the device plugin did its job. The runtime path did not. Almost always the pod spec omits runtimeClassName: nvidia, so the container started on the default runtime, which knows nothing about GPUs and injected nothing.
This is the failure worth internalising, because every instinct points the wrong way. The pod is Running, so people check the application. The node has GPUs, so people check capacity. The resource request was honoured, so people check the driver. The actual cause is a single missing line in the spec, and the tell is that scheduling and runtime are separate decisions made by different components.
The related failure. If the pod does not start at all and the error mentions an unknown runtime handler, the RuntimeClass names a handler the container runtime has never heard of. A RuntimeClass object is a pointer, and it resolves to a handler configured in containerd. Installing the operator does not configure your node's containerd handler list, which is why the runtime has to be registered before anything references it.
Symptom: everything works, then stops after a node reboot
What you see. A cluster that was fine yesterday has nodes with no GPU resources today, usually after a kernel update or a reboot.
What it means. The driver stack did not come back. Either the driver was built against a kernel that has since been replaced, or a component that depends on it started before it was ready and gave up.
Why it looks like an operator problem and is not. The chain is intact; its foundation moved. This is the case for pinning kernel versions on GPU nodes and for treating a driver upgrade as a planned change rather than something that rides along with routine patching.
A diagnostic order that works
Walk it in the order the chain runs, and stop at the first thing that is wrong.
nvidia-smion the host. Is there hardware with a working driver?kubectl describe node. Are the labels there? Isnvidia.com/gpuallocatable?- If both are fine, is the pod Pending or Running? Pending is a scheduling question, Running is a runtime one.
- If Pending, read the full message. Insufficient resource, untolerated taint, and no preemption victims are three different problems.
- If Running, check for
runtimeClassNamein the spec before anything else.
The value of the order is that it stops you investigating a component that could not possibly have run yet. Most time lost on these incidents goes to checking things downstream of the actual break.
Setting it up so these happen less
Three habits reduce the frequency.
Register the runtime before you install the operator. A RuntimeClass that points at a handler containerd does not have fails at container creation, and the error appears far from the install that caused it.
Pin what the driver depends on. Kernel upgrades on GPU nodes deserve a change window. The chain is only as stable as the driver underneath it.
Make the labels part of your manifests. A workload that targets nvidia.com/gpu.product explicitly fails loudly on a fleet where feature discovery has stopped, rather than landing quietly on the wrong hardware. Loud failures are cheaper than silent misplacement.
Where to learn this properly
Reading symptom lists gets you part of the way. The rest comes from having caused each failure yourself, which is why the AI Infrastructure Engineer course has a triage day for exactly this: three GPU pods, each broken at a different link, with no indication of which is which. The course then does the same for the pod lifecycle more broadly, and ends in a capstone where a cluster is failing three unrelated ways at once.
For the full sequence this sits inside, see the AI infrastructure engineer roadmap. For the career context, how to become an AI infrastructure engineer covers the routes in and how these interviews are run, which is almost always as a triage exercise like the ones above.