Step 1: The controller pattern: why bare pods aren't production
The controller pattern — why bare pods aren't production
A bare Pod is a single container's life. The kubelet runs it; when its container exits, the pod stays in Succeeded (exit 0) or Failed (non-zero) — but it does not restart unless you explicitly write restartPolicy: Always, and even then the pod doesn't come back if it gets evicted, deleted, or its node disappears. There's nothing reconciling "I want one of these running" — the bare Pod is a one-shot.
Production AI workloads need a controller that reconciles desired state:
- "Keep 3 replicas of this inference server Running."
- "Run exactly one DCGM exporter on every GPU node."
- "Maintain 4 trainer pods with stable hostnames
trainer-0throughtrainer-3, started in order."
Three controllers cover ~95% of these patterns, and the AI workload shape determines which one:
| Controller | Shape | Used for in AI |
|---|---|---|
Deployment | N stateless replicas, random pod names, scale up/down freely, rolling updates | Inference servers, API gateways, model proxies, anything that scales horizontally |
StatefulSet | N replicas with stable hostnames, ordered creation/deletion, per-pod PVCs | Distributed training (PyTorch DDP, Horovod), parameter servers, anything where peers must address each other by name |
DaemonSet | One pod per matching node, automatic scheduling on new nodes | DCGM exporter, device plugin, GFD, node-local model caches, log/trace agents |
A fourth controller, Job, runs N pods to completion (good for one-off training runs), and CronJob schedules Jobs — covered briefly in the FAQ but not the focus of this lab.
Read what's already running
The lab platform's own infrastructure includes a couple of controllers in kube-system. Run the step's command (or click Run) — it will:
- Apply a bare
lonely-podand watch it terminate (no controller, no restart). - List every Deployment, StatefulSet, and DaemonSet in the cluster.
# What controllers exist in this cluster right now
kubectl get deploy,sts,ds -A
You'll see kube-system Deployments for the cluster's own infrastructure (CoreDNS, etc.). You won't see any user controllers — you'll be authoring each one in the next three steps.
How a Deployment works under the hood
A Deployment doesn't manage Pods directly — it manages a ReplicaSet, which manages Pods:
Deployment (you author this)
│
↓ owns
ReplicaSet (auto-created by Deployment controller)
│
↓ owns
Pods (auto-created by ReplicaSet controller, named <rs>-<random>)
When you kubectl apply a new Deployment image, a new ReplicaSet is created with the new template; the old ReplicaSet is scaled to 0. That's how rolling updates work — old RS → new RS, with overlap controlled by strategy.rollingUpdate. Reading the chain kubectl describe deploy → kubectl get rs → kubectl get pods is one of the most useful diagnostics on the platform.
How a StatefulSet differs
StatefulSet manages Pods directly (no intermediate ReplicaSet) and gives each one a stable name and identity — trainer-0, trainer-1, ... trainer-(N-1). The numbers are sequential, never randomly suffixed. The pods are created in order (0 → 1 → 2 ...) and each must be Ready before the next is started. Termination on scale-down is the reverse order (highest-numbered first). Each pod can have its own PVCs auto-created via volumeClaimTemplates — data-trainer-0, data-trainer-1, etc.
This shape is exactly what distributed training frameworks need: every rank knows the others' hostnames a priori, rank 0 starts first as the coordinator, and per-rank checkpoint volumes don't collide.
How a DaemonSet differs
DaemonSet doesn't have replicas — it has a target set of nodes (matched via nodeSelector or nodeAffinity). It runs one pod per matching node and automatically schedules a new pod when a matching node joins the cluster (or removes a pod when a node leaves). The status fields show this:
NUMBER-DESIRED NUMBER-CURRENT NUMBER-READY NUMBER-AVAILABLE
3 3 3 3
If a DaemonSet's nodeSelector doesn't match any node, you get 0/0 — the controller didn't fail, it just has nothing to schedule. That's a common "missing pods" diagnostic.
Why this matters in practice: picking the wrong controller is one of the most common production mistakes — and the symptoms are subtle ("my distributed trainer can't find its peers" usually means someone used Deployment and the pods got random names). The NCA-AIIO exam tests the workload-shape-to-controller mapping directly. The next four steps make each one concrete.
