Preporato

hands-on labon a real NVIDIA GPU

GPU Container Lifecycle: Build, Test, Ship, Rollback

Walk through the full lifecycle of a production GPU container, multi-stage Dockerfile, self-hosted GPU CI, a fail-fast smoke test, and a Kubernetes Deployment with readiness probes gated on real GPU compute.

time
40 min
checked steps
4
level
Intermediate
setup
None
NCP-AIONCA-AIIONCA-GENLNCP-AIIPart of the AI Engineer Course

$29.99a month with Pro

Pro opens this lab and every other one. Cancel any time.

4 steps, each checked when you finish it

A check runs your work at the end of every step. Hints and the full solution are there if you get stuck.

  1. 1

    The production Dockerfile

    read this step below ↓

  2. 2

    The CI workflow

  3. 3

    The GPU smoke test

  4. 4

    Kubernetes rollout + rollback

Step 1 as it appears in the lab

The lab’s own text. The hint and the solution stay inside the lab.

Step 1: The production Dockerfile

Every rule below is in the Dockerfile for a reason:

  • Multi-stage build — builder compiles/installs; the final stage is a slim runtime. Saves 2-5 GB of image size. Critically, it prevents leaking build-time secrets (pip index URLs with tokens).
  • nvidia/cuda:X.Y.Z-runtime-ubuntu22.04, not -devel. Runtime has the loader + libcudart + libcudnn. Devel adds nvcc, nvrtc, headers — ~2 GB extra. You don't compile CUDA at runtime.
  • Pin every version. torch==2.3.0, base image with full X.Y.Z, --no-install-recommends. Unpinned latest means yesterday's build is a different image than today's — catastrophic for reproducibility.
  • Non-root USER. If the container is compromised, the attacker gets appuser, not root. Kubernetes SecurityContext can also enforce this, but defense in depth.
  • HEALTHCHECK invokes the GPU smoke test, not a bare HTTP endpoint. An HTTP endpoint that returns 200 on a broken GPU is worse than no endpoint.

Spot the bug

There's a common production bug encoded below: the builder stage installs via pip install --user, then the final stage copies /root/.local → /home/appuser/.local. The appuser account needs to own those files to import them — --chown=appuser:appuser on the COPY is easy to forget, and forgetting it leaves the app broken only when an unprivileged user tries to run it (your prod). Notice we include that --chown explicitly — this is the pattern you want to internalize.

from IPython.display import Code

# The production Dockerfile — now a real file under workspace/ (CI, IDE linters,
# and docker build can all use it directly). We load it here so the check_script
# can validate its contents.
dockerfile_path = '/home/labuser/workspace/Dockerfile'
with open(dockerfile_path) as f:
    dockerfile = f.read()
dockerfile_lines = len(dockerfile.splitlines())
print(f"Dockerfile: {dockerfile_lines} lines, "
      f"{dockerfile.count('FROM ')} stages")

Code(filename=dockerfile_path, language='docker')

what you get

  • A JupyterLab notebook on a real NVIDIA GPU
  • A check at the end of every step
  • Hints and the full solution
  • Nothing to install or configure

before you start

  • Comfortable with Docker multi-stage builds and Dockerfile directives
  • Basic Kubernetes (Deployments, probes, nodeSelector)
  • Familiarity with CI/CD concepts (GitHub Actions or equivalent)

Exam domains covered

GPU Infrastructure & OperationsModel Deployment & Inference Optimization

Part of the GPU and Kubernetes track

Lab 9 of 20. GPU scheduling on your own Kubernetes cluster, the GPU Operator, CUDA kernels and profiling.

See the whole track
Beginner · 5 minWelcome to NCA-AIIO Labs: Schedule Your First GPU PodKubernetesPro
Intermediate · 40 minNVIDIA GPU Operator on k3s: Single-Node Kubernetes for GPU WorkloadsNVIDIA GPUPro
Intermediate · 35 minInside the NVIDIA GPU Operator: From Helm to Workload-ReadyKubernetesPro
Intermediate · 40 minKubernetes Resource Requests & Limits: Who Gets What, and Who SurvivesKubernetesPro
Intermediate · 35 minMulti-GPU-Type Targeting: nodeSelector, nodeAffinity & TolerationsKubernetesPro
Intermediate · 35 minPriorityClass & Preemption: Who Survives the GPU SqueezeKubernetesPro
Intermediate · 35 minWorkload Controllers: Deployment, StatefulSet, DaemonSet for AIKubernetesPro
Intermediate · 35 minPersistent Storage for AI Workloads: PVCs, StorageClass & the Checkpoint PatternKubernetesPro
this labGPU Container Lifecycle: Build, Test, Ship, RollbackNVIDIA GPUPro
Intermediate · 35 minRolling Updates, Rollback & Blue-Green for AI InferenceKubernetesPro
Intermediate · 40 minStuck-Pending Triage Day: Diagnose Any GPU Pod That Won't RunKubernetesPro
Intermediate · 40 minGPU Observability: From nvidia-smi to a Production Monitoring StackNVIDIA GPUPro
Advanced · 50 minGPU Health Checks + Auto-RemediationNVIDIA GPUPro
Advanced · 45 minCUDA Programming FundamentalsNVIDIA GPUPro
Intermediate · 35 minProfile PyTorch Training with the Built-in ProfilerNVIDIA GPUPro
Intermediate · 35 minNsight Systems Profiling: Finding the Bottleneck That Costs You 40% of Your GPUNVIDIA GPUPro
Advanced · 45 minGPU Sharing: Streams, MPS, MIG, and the Real Cost of ContentionNVIDIA GPUPro
Intermediate · 30 minNVIDIA DALI: GPU-Accelerated Data PipelinesNVIDIA GPUPro
Intermediate · 40 minGPU-Accelerated Data Science with RAPIDSNVIDIA GPUPro
Intermediate · 35 minGPU Cost & Efficiency AuditNVIDIA GPUPro

Questions about this lab

Why target runs-on: [self-hosted, gpu] instead of a GitHub-hosted runner?

GitHub-hosted runners don't have NVIDIA GPUs, your docker run --gpus all test step would either skip or silently pass. You either register a self-hosted runner on a GPU host, use a managed GPU CI provider (BuildJet, Actuated, Namespace), or rely on GitHub's large-runner GPU tier where available. The lab shows the self-hosted, gpu pattern because it's the most portable and because it teaches you to separate the 'where does my image build' question from the 'where does my image test' question, in production those often run on very different hardware.

Should the HEALTHCHECK in the Dockerfile call curl http://localhost:8000/healthz?

No, that's the mistake this lab is specifically designed to correct. A successful HTTP 200 from your server process proves Python is running; it proves nothing about the GPU. Dockerfile HEALTHCHECK should invoke a GPU-aware probe, python3 /app/smoke_test.py that calls torch.cuda.is_available(), initializes NVML, and allocates a small tensor, so Docker marks the container unhealthy when the driver, the toolkit, or the card itself fail. Your Kubernetes readiness probe reuses the exact same script.

Why distinguish readiness from liveness when both call the same smoke test?

Readiness answers 'should this pod receive traffic right now?' and failure simply pulls the pod out of the Service endpoint list, no restart. Liveness answers 'is this pod permanently wedged?' and failure triggers a kubelet restart. Using the same script is fine; using the same thresholds is a bug. Readiness should fail fast (a few seconds) during startup while the model warms up; liveness should fail slow (tens of seconds, with a tolerant failureThreshold) so transient Xid errors or a briefly-stuck kernel don't thrash your pods into a CrashLoopBackOff.

What can go wrong at runtime that CI and HEALTHCHECK won't catch?

Plenty. Xid fatal errors after hours of load. Memory fragmentation that only surfaces when a larger-than-usual batch arrives. Thermal throttling that turns into a kernel timeout. cuDNN version drift between warm-up and real traffic. Driver hangs after a specific sequence of CUDA API calls. Model shard OOMs under production-shaped inputs. CI runs briefly on a cold GPU; real traffic is long, hot, and adversarial. That's exactly why the livenessProbe reruns the smoke test continuously and why you'd graduate to a richer probe that exercises a real model forward pass.

Why use maxUnavailable: 0, maxSurge: 1 for the rolling update?

GPU pods are expensive and cluster GPU capacity is usually the binding constraint. maxUnavailable: 0 guarantees the old pod keeps serving traffic until the new one passes readiness, so you never drop below full capacity. maxSurge: 1 says you're willing to temporarily run one extra replica during the rollout, critical because without surge, with maxUnavailable: 0, the rollout would deadlock waiting for a pod that can't start until another pod dies. The combination gives you zero-downtime deploys on constrained GPU nodes.

How does grading work for the Kubernetes step?

The validator parses your deployment_yaml string and checks for the required fields: apiVersion, kind: Deployment, strategy.rollingUpdate with both maxUnavailable and maxSurge, nvidia.com/gpu in resource limits, and both readinessProbe and livenessProbe. It also inspects your rollback_commands list for kubectl rollout history and undo, and verifies your image_promotion_flow defines at least three stages (dev → staging → prod) each with env, tag_pattern, and gate keys. Nothing is applied to a live cluster, the lab grades the artifacts, not a running rollout.

What you'll build in this GPU container lifecycle lab

Shipping a GPU container to production is where most AI teams first discover that docker build + docker push is not a deploy pipeline. This lab gives you the full pattern, multi-stage Dockerfile, self-hosted GPU CI, a GPU-aware smoke test, Kubernetes Deployment with probes gated on real compute, and a rollout/rollback playbook, so the next time a driver mismatch, a cuDNN drift, or a silent VRAM leak tries to reach users, it hits a wall instead. You'll leave with a runnable Dockerfile on nvidia/cuda:*-runtime-ubuntu22.04, a GitHub Actions workflow targeting a self-hosted, gpu runner, a three-check smoke test with distinct exit codes, a Deployment YAML with maxUnavailable: 0 / maxSurge: 1 and both probes, and a concrete mental model of why four layers of essentially-the-same-test is the feature, not the bug. ~40 minutes on a real NVIDIA GPU pod we hand you; no local Docker, no kubectl context juggling.

The technical backbone is defense in depth on the same question, can this container actually use a GPU right now?, asked from four places. CI runs the smoke test on a self-hosted GPU runner (GitHub-hosted runners have no GPUs, so docker run --gpus all silently no-ops and your pipeline lies to you). Dockerfile HEALTHCHECK invokes python3 /app/smoke_test.py, NVML init, torch.cuda.is_available(), tensor round-trip, not curl /healthz, because HTTP 200 proves Python is running, not that CUDA is. The readiness probe reuses the same script to gate traffic on a pod that booted but can't allocate VRAM. The liveness probe reruns it continuously to catch the slow-motion failures, Xid fatal errors, thermal throttling, memory fragmentation, cuDNN version drift, kernel hangs, that CI on a cold GPU in a short run will never see. maxUnavailable: 0 + maxSurge: 1 gives you zero-downtime rolling updates on GPU-constrained clusters without deadlocking.

Prereqs: Docker multi-stage builds, basic Kubernetes (Deployments, probes, nodeSelector, resources.limits), and CI/CD concepts (GitHub Actions or equivalent). Preinstalled on the lab pod: Docker, NVIDIA Container Toolkit, kubectl, PyTorch, and CUDA. Grading checks the artifacts the way a reviewer would: the Dockerfile must have ≥2 FROM stages, a non-root USER, and a GPU-aware HEALTHCHECK; the workflow must declare build + test jobs on a GPU runner with a push stage gated on main; the smoke test must exit 0 on healthy and non-zero when CUDA_VISIBLE_DEVICES=''; the Deployment must declare both probes, nvidia.com/gpu, and the rolling-update knobs. The reflection step asks you to instrument past the four layers, DCGM DCGM_FI_DEV_XID_ERRORS, SM_ACTIVE, a model-shaped forward-pass probe, which is how you graduate from 'ships clean' to 'stays clean at 3am'.