TL;DR: Seven stages, in an order that matters because each one explains a class of failure the next assumes you understand. Kubernetes resource management seen through AI workloads, then the GPU operator chain, scheduling and sharing, storage and rollouts, serving capacity, observability, and cost. Learn each stage by breaking something on a real cluster rather than by reading about it, and finish each one with an artifact you can describe from memory. Twelve to fifteen weeks of evenings is a realistic pace.
Most GPU infrastructure knowledge is acquired in the worst possible order: an incident happens, you learn the one component involved, and you move on with a mental model made of unconnected facts. That works until two components fail at once and you have no framework for deciding which to suspect.
This roadmap is the other order. Each stage builds the vocabulary the next stage's failures are described in, so by the end a symptom narrows to a subsystem instead of a guess. It is the sequence the AI Infrastructure Engineer course teaches, and every stage names what to build so the learning leaves evidence.
Read this alongside
This is the technical sequence. For the career side, including the three entry routes and how these interviews are actually run, read how to become an AI infrastructure engineer.
What the role actually is
An AI infrastructure engineer runs the GPU clusters that everyone else's models depend on. The job sits where Kubernetes platform engineering meets GPU operations, and the reason it is hard to hire for is that most candidates have one half convincingly and the other half not at all.
The daily work is placement, contention, durability, capacity and cost. The daily failure is a workload that will not run and an object graph in which the thing showing the symptom is almost never the thing at fault.
Stage 0: prerequisites you should already have
You need Linux comfort, containers as a concept rather than a mystery, and enough Kubernetes to know what a pod and a node are. You do not need driver experience, GPU programming, or any machine learning background. Nothing below assumes you can write a training loop, only that you can recognise one when it is failing.
If you have never run kubectl against a cluster you were responsible for, spend a week on that first. Everything here compounds on it.
Stage 1: Kubernetes resource management, through AI workloads
The most consequential six lines in any manifest are requests and limits, and almost everyone learns them slightly wrong.
Requests are what the scheduler reserves, and they are the only number it reads when deciding whether a pod fits. Limits are what the kubelet enforces at runtime: exceed a CPU limit and you are throttled, exceed a memory limit and you are killed. Quality of service class falls out of how the two relate, and it decides who gets evicted when a node runs short.
Then the rules that make GPUs different. They come in whole integers, they cannot be overcommitted, and the request is forced equal to the limit. Two pods can share a CPU core by overcommitting; they cannot share a GPU device that way, which is why sharing needs a deliberate mechanism later in the roadmap.
Finish the stage by picking the right controller for each shape: a Deployment for inference where replicas are interchangeable, a StatefulSet for distributed training where each replica needs identity and storage, and a DaemonSet for per-node infrastructure where the count should follow the cluster rather than a number you maintain.
Build: a namespace with a LimitRange and a ResourceQuota, and three workloads that each fail differently when you get the resource stanza wrong.
Stage 2: the GPU operator chain
This is the stage that turns GPUs from magic into components, and it pays for itself the first time you triage.
A GPU becomes usable through a chain. Feature discovery inspects the hardware and publishes node labels, which are a scheduling API rather than decoration. The device plugin advertises the allocatable resource so the scheduler has something to satisfy. A RuntimeClass selects the container runtime that injects driver libraries and device nodes into the container. Telemetry components expose what the card is doing.
Learn to read a node and say which link is missing. The single most confusing symptom in this field is a pod that schedules happily and then finds no GPU inside itself, and it means the resource existed while the runtime path did not.
Build: a single-node cluster where you installed the operator by hand, plus notes attributing each node label and each allocatable resource to the component that produced it. When something in that chain breaks later, GPU operator troubleshooting walks the symptoms link by link.
Stage 3: scheduling and sharing
Placement first, contention second.
Placement across a mixed fleet uses four primitives with different semantics. Node selectors are simple equality. Node affinity adds richer operators, set membership and soft preferences. Taints are the node-side opposite: they exclude by default and admit only what tolerates them, which is how you reserve a pool rather than merely attract work to it.
Contention is priority and preemption, the only mechanism Kubernetes offers for resolving a full cluster with intent. A priority class has three fields and each does something distinct: the value orders workloads, the preemption policy decides whether that order is acted on by evicting anyone, and the global default decides who inherits the class without asking. That third field causes more production incidents than the other two combined, because it silently rewrites the priority of everything that never named a class.
Then sharing one card four ways: streams inside a process, time-slicing, MPS, and MIG. Only one of them gives two tenants a real memory and fault boundary, and the comparison covers which and why.
Build: placement manifests for a mixed fleet that defend their rules, and a demonstration of preemption you can explain from the events.
Stage 4: storage and rollouts
Two failure modes that cost more than they should.
Storage first: a pod's filesystem is ephemeral, so a checkpoint written inside the container dies with it. A PersistentVolumeClaim fixes that, and the details matter more than they look. Access modes are the trap, because ReadWriteOnce is scoped to a node rather than to a pod. The reclaim policy decides whether your data outlives the claim. Volume binding mode and node affinity together decide whether a volume can follow a pod that gets rescheduled somewhere else.
Rollouts second: shipping a new model version without dropping requests. A rolling update is safe because of the readiness probe, which keeps an unready pod out of the Service endpoints and stalls the rollout rather than tearing down healthy replicas. Rollback is a command. Blue-green is a selector swap, atomic because both versions are already running.
Build: a checkpointing setup that survives a node drain, which is best learned by first watching it fail, and a bad rollout you abort and revert.
Stage 5: serving capacity
The stage that connects the cluster to a number somebody promised a customer.
Batching is the single technique that changes throughput most, and it comes with a knob that trades latency for it. Build a batcher by hand once, with a maximum batch size and a queue delay, so the trade is something you have measured rather than read about. Then sweep the batch size against a latency objective and notice that the answer is a constrained maximum rather than a peak.
Move to a production inference server and learn what it does differently: managing its cache in blocks so more sequences fit, and keeping the batch full by admitting new work as finished sequences leave rather than waiting for the slowest member. Finish by sizing for the hardware you actually have, sweeping batch size against numeric precision to name an operating point.
Build: an inference deployment with a measured operating point and the sweep that justifies it.
Stage 6: observability and profiling
You cannot manage contention or cost without measurement, and the measurement has to be at the right layer.
Start at the driver: utilisation, memory, power, per-process allocation. Sample over time rather than once, because the shape carries the diagnosis. A regular sawtooth means the card is being starved between batches, and the average hides it completely.
Build a watchdog next. The classic production failure is a leaked process still holding memory after its job is gone, which makes a free card look full to everyone else. A remediator that kills the process is half the job; the half people skip is re-probing to confirm the memory actually came back.
Then two profilers for two questions. A framework profiler attributes time and memory to individual operators, because it sits inside the framework and knows their names. A system profiler shows the whole timeline across host and device, which is what you need when the question is whether the GPU was waiting rather than which operator was slow.
Build: a metrics endpoint your existing monitoring can scrape, and a profile that found a real bottleneck you then fixed.
Stage 7: cost
The stage that makes the role legible to people who approve budgets, and the one most engineers skip.
The pipeline has four parts. Measure, which you can already do. Classify, which labels each second as useful work or waste and is what turns a utilisation number into a diagnosis. Price, which attaches dollars. Recommend, which names a specific workload, a specific action and the money attached to doing it.
The fourth part is what separates a report from a recommendation. "The fleet averages thirty-four percent" is true and leaves the reader with nothing to do. "Move this job to a shared slice and save four thousand a year" gets scheduled.
Build: the four-stage audit, run against a real workload, ending in a recommendation with a number.
Where certifications fit
Two are aimed squarely at this ground. NVIDIA's AI Infrastructure and Operations associate exam covers stages one through four closely, and the professional tier reaches into the observability and lifecycle work of stages six and seven.
Treat them as deadlines that structure study rather than as the qualification. Nobody is hired for reciting the operator chain. People are hired for having run one, and the certificate is a filter-passing signal that gets your artifacts read.
How long it takes
Twelve to fifteen weeks of consistent evenings is realistic from a platform or SRE background, and the distribution is uneven. Stages one and two take the longest because they build the vocabulary. Stages three and four go faster once the operator chain makes sense. Stages five through seven are quicker still, and they are where the portfolio artifacts come from.
Coming from machine learning rather than platform work, add a few weeks at the start for cluster fundamentals. Coming from systems administration, expect stages one and four to be the unfamiliar ones and the GPU layer to feel comparatively natural.
The pace that fails is reading all seven stages and building none of them. The pace that works is one artifact per stage, however small.
Next steps
The AI Infrastructure Engineer course runs these seven stages across twenty-one labs on real clusters and GPU pods, and ends in a capstone that hands you a cluster failing three ways at once. If you are still choosing between this and the pipeline side of the work, AI infrastructure engineer versus MLOps engineer draws the line by what each role owns when the pager goes off.