Step 1: The ephemeral storage problem: why pods need PVCs
The ephemeral storage problem
A pod's filesystem is ephemeral by default. Anything written to /tmp, /data, /checkpoints, or any other path inside the container lives in the container runtime's writable layer — a copy-on-write overlay on top of the image. When the pod is deleted, the layer is destroyed. When the container restarts inside an existing pod (e.g., from a crash), the writable layer is also discarded; you come back to the image's original state.
This is fine for stateless workloads (HTTP servers, batch jobs that read input and write output to S3). It is catastrophic for AI workloads — model checkpoints, dataset caches, embedding indices, fine-tuning state, all gone when something restarts.
The fix is PersistentVolumeClaims (PVCs). A PVC is a request for storage that outlives the pod. The cluster matches the request to a PersistentVolume (PV), the PV mounts a real disk somewhere, and the pod mounts the PV at a path of your choosing.
The three objects you'll work with
| Object | Authored by | What it represents |
|---|---|---|
StorageClass (cluster-scoped) | Cluster admin | A recipe for provisioning storage — names a CSI driver and its parameters. Persistent across pods. |
PersistentVolume (PV, cluster-scoped) | Usually auto-created by the StorageClass's provisioner | An actual disk — backed by a hostPath, a network volume, a cloud disk. |
PersistentVolumeClaim (PVC, namespaced) | Pod author (you) | A request for storage — size, accessMode, optional StorageClass. The pod references it by name. |
The lifecycle:
1. You write a PVC: "I need 5Gi RWO from default StorageClass"
2. Cluster sees the PVC; consults the named StorageClass's provisioner
3. Provisioner creates a PV (a real disk) of that size
4. PVC.status.phase: Pending → Bound (linked to the PV)
5. Your pod's spec.volumes[].persistentVolumeClaim.claimName = <pvc-name>
6. Kubelet on the chosen node attaches the volume + mounts it inside the container
7. Pod runs; reads/writes to the mount path are durable
Read the cluster
Run the step's command (or the Run button) to see what's available. The platform pre-installed one StorageClass (local-path) — that's what we'll use throughout the lab.
# 1. All StorageClasses with their provisioners and key fields.
kubectl get storageclass -o wide
# 2. PVCs in this namespace (should be empty — fresh start).
kubectl get pvc
# 3. The "default" StorageClass — used when a PVC omits storageClassName.
kubectl get sc -o jsonpath='{range .items[*]}{.metadata.name}: {.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}{"\n"}{end}'
Anatomy of the StorageClass you have
The output of kubectl get sc -o wide has six columns that every NCA-AIIO student should be able to interpret on sight:
| Column | What it means | Local-path's value |
|---|---|---|
NAME | StorageClass name; PVCs reference it via storageClassName | local-path |
PROVISIONER | The CSI driver that creates PVs for this class | rancher.io/local-path |
RECLAIMPOLICY | What happens to the PV when the PVC is deleted: Delete (destroy the disk) or Retain (preserve the data, leave the PV in Released) | Delete |
VOLUMEBINDINGMODE | When does the PV get provisioned: Immediate (right when the PVC is created) or WaitForFirstConsumer (defer until a pod references the PVC, so the scheduler can pick a node first) | WaitForFirstConsumer |
ALLOWVOLUMEEXPANSION | Can a PVC be resized after creation? | false for local-path |
AGE | Object age | varies |
Two of these are pedagogically critical:
reclaimPolicy: Delete — when you kubectl delete pvc, the underlying disk + data is destroyed. For training checkpoints in a real production cluster, you'd typically want Retain (keep the data) plus a backup mechanism (snapshots, copy-to-S3-on-finalize). For lab work, Delete is fine.
volumeBindingMode: WaitForFirstConsumer — when you kubectl apply -f pvc.yaml, the PVC will sit Pending and produce a "WaitForFirstConsumer" event. This is normal. It only binds when a pod references the PVC. The reason: the scheduler picks the pod's node first, then the provisioner creates a PV that's local to that node. If the PV were created up front, the pod might be unable to schedule on the node where the PV lives.
A common newcomer mistake: applying a PVC, seeing it Pending, panicking. With WaitForFirstConsumer, "Pending until a pod claims it" is correct behavior. We'll see this in the next step.
Why this matters in practice: every NCA-AIIO student has seen a colleague's training run lose hours of progress to ephemeral storage. The PVC model is the answer, but only if you understand the StorageClass parameters that control durability (reclaimPolicy) and binding timing (volumeBindingMode). The rest of this lab makes those parameters concrete.
