Stuck-Pending Triage Day: Diagnose Any GPU Pod That Won't Run

The five-gate pod lifecycle and where pods get stuck

Every pod has to clear five gates between kubectl apply and answering its first request. A pod that's "stuck" is wedged at exactly one of them. Knowing which gate is the difference between fixing it in 30 seconds and chasing the wrong layer for 30 minutes.

kubectl apply ─► [1. Admission ]─► [2. Scheduling ]─► [3. Image pull ]─► [4. Runtime startup ]─► [5. Readiness ]─► serving
                  API server       kube-scheduler       kubelet              kubelet                kubelet
                                                        (CRI client)         (CRI runtime)          (probes)

                  rejected:        Pending              ImagePullBackOff     ContainerCreating      Running 0/1
                  Forbidden        FailedScheduling     ErrImagePull         (CrashLoopBackOff       (Ready never
                                                                              if container exits)    flips True)

The five gates

1. Admission — the API server validates the request

The API server runs every PodSpec through admission controllers before it's even persisted to etcd. Common rejections:

• ResourceQuota — Forbidden: exceeded quota: requests.cpu, requested: 5, used: 1, limited: 2
• LimitRange — Forbidden: maximum cpu usage per Container is 2, but limit is 5
• Pod Security Standards — Forbidden: hostNetwork=true is not allowed
• Kyverno / OPA admission policies — image registry not in allowlist

Diagnostic: kubectl apply returns the rejection synchronously. The pod never appears in kubectl get pods because it was rejected before persistence. If you don't see your pod, the bug is here.

2. Scheduling — kube-scheduler picks a node

The pod exists in etcd with .spec.nodeName: "". The scheduler walks the node list looking for one that satisfies:

• requested resources (nvidia.com/gpu, cpu, memory) ≤ node Allocatable
• nodeSelector and nodeAffinity match the node's labels
• pod's tolerations cover the node's taints
• podAntiAffinity doesn't conflict with already-placed pods

Diagnostic: Pod is Pending, .spec.nodeName is empty. kubectl describe pod <name> Events section has a Warning FailedScheduling event with the specific reason: "0/1 nodes are available: 1 Insufficient nvidia.com/gpu", "didn't match Pod's node affinity/selector", "node(s) had untolerated taint X".

3. Image pull — kubelet downloads the container image

Once scheduled, the kubelet is responsible for the rest. First step: pull the image (skipped if already present, cached by digest).

• Tag doesn't exist → 404 Not Found → ErrImagePull → ImagePullBackOff
• Private registry, no creds → 401 Unauthorized → ditto
• Docker Hub rate limit → 429 Too Many Requests → ditto
• DNS / firewall → dial tcp ... connection refused

Diagnostic: Pod is ContainerCreating (or directly ImagePullBackOff after several failed pulls). .spec.nodeName is set. kubectl describe pod Events show Failed to pull image "X": ... with the underlying HTTP/network error. kubectl logs returns nothing because no container has started yet.

4. Runtime startup — CRI runtime creates the container

Once the image is local, the kubelet asks the container runtime (containerd / CRI-O) to create the sandbox and the container. Failures here:

• runtimeClassName references unknown handler → failed to create pod sandbox: ... no runtime for "nvidia" is configured
• securityContext violations → runAsNonRoot set but image runs as root
• PVC not bound → MountVolume.SetUp failed for volume "X": persistentvolumeclaim "X" not found
• OCI hook failures (NVIDIA Container Runtime, AppArmor, SELinux) → various per-runtime errors

Diagnostic: Pod is ContainerCreating indefinitely. kubectl describe pod Events show "Failed to create pod sandbox" or "MountVolume.SetUp failed". The events are emitted by the kubelet, NOT the scheduler.

5. Readiness — probes gate the Ready=True transition

The container is running, but it isn't Ready until its readinessProbe passes. Until Ready=True, the Service excludes it from Endpoints. Failures here:

• No probe → Ready=True immediately (often wrong; see the rolling-update lab)
• Probe timeout → Readiness probe failed: connection refused
• Probe path wrong → Readiness probe failed: HTTP probe failed with statuscode: 404

Diagnostic: Pod is Running 0/1. kubectl describe pod Events show Warning Unhealthy Readiness probe failed: .... Container itself is alive — kubectl logs works.

The 30-second triage flowchart

$ kubectl get pod <name>

  Not in list?              → Gate 1 (Admission). `kubectl apply` returned the error;
                              re-apply and read the kubectl output.

  STATUS: Pending           → Gate 2 (Scheduling). describe pod → Events → FailedScheduling.

  STATUS: ContainerCreating → Gate 3 or 4. describe pod → Events:
       message has "pull" → Gate 3 (Image)
       else                → Gate 4 (Runtime / volume / sandbox)

  STATUS: ImagePullBackOff  → Gate 3.

  STATUS: Running but 0/1   → Gate 5 (Readiness). describe pod → Events → Unhealthy.

  STATUS: CrashLoopBackOff  → Gate 4-5 boundary. Container started and crashed.
                              → kubectl logs <name> --previous (the last crashed container's logs)
                              → describe pod for OOMKilled signals

Read your lab cluster's facts

Run the step's command (or click Run). It dumps the cluster's Allocatable, namespace quota, RuntimeClasses, StorageClasses, and PriorityClasses. These are the facts you'll use to recognize what should be in the next four steps' broken pods.

Why this matters in practice: the gate framework is universal — it applies to every pod on every Kubernetes cluster, GPU or not. NCA-AIIO frames triage questions as "given this kubectl output, name the gate" — and once you can place the gate, the fix follows directly.