TL;DR: Four mechanisms, and they are not alternatives to each other. CUDA streams overlap work inside one process. Time-slicing lets several processes take turns, which costs each of them roughly half its speed and buys no throughput. MPS funnels several processes through one GPU context so they genuinely run at once, with no memory isolation between them. MIG partitions the card in hardware, giving real isolation on A100 and H100 class parts at the cost of fixed geometry. Pick by asking whether you need concurrency, isolation, or both, because a fleet sitting at fifteen percent is usually one bad answer to that question.
A GPU costs the same whether it is fifteen percent busy or ninety-five percent busy. Most fleets sit near the first number, and the reason is rarely that the workloads are small. It is that small workloads were each given a whole card, because the alternative looked complicated and the person deciding did not have a model for choosing between the options.
The options are not interchangeable. Two of them give you concurrency, one gives you isolation, one gives you neither, and the most common mistake in production is reaching for the one that gives neither because it is the easiest to configure.
Measure this yourself
The AI Infrastructure Engineer course has you benchmark all four on a real GPU, including the contention penalty below, then write the Kubernetes device-plugin configuration for each strategy.
The four mechanisms at a glance
| Mechanism | Scope | Real concurrency | Memory isolation | Fault isolation |
|---|---|---|---|---|
| CUDA streams | Inside one process | Yes, when SMs are spare | No | No |
| Time-slicing | Across processes | No, they take turns | No | Separate contexts |
| MPS | Across processes | Yes | No | No |
| MIG | Across processes | Yes | Yes | Yes |
Read that table by column rather than by row. The question you are answering decides which row you want, and most teams pick a row before deciding on a question.
CUDA streams: overlap inside one process
A stream is an ordered queue of kernel launches. Work on two different streams may run at the same time if the hardware has spare streaming multiprocessors, and work on the same stream runs strictly in order. The default stream serialises everything, which is fine for a script and wrong for a server handling several requests at once.
The important limit is that overlap requires idle capacity. One large matrix multiply occupies the whole card for its duration, so a second stream has nothing to run on. Overlap shows up when kernels are small, memory-bound, or launch-bound, which is exactly the profile of inference serving.
Measured on a consumer card, putting two independent chains on separate streams instead of the default gives roughly a 1.8x speedup. Grow the matrices until they saturate the SMs and the two streams converge back to serial, which is the clearest demonstration that this mechanism is about filling gaps rather than creating capacity.
Streams are not a multi-tenancy tool. Everything shares one process and one address space, so there is no boundary of any kind between the work.
Time-slicing: the default, and the tax nobody measures
Run two processes against one GPU with no sharing strategy configured and they both work. That is what makes this the trap.
They do not run concurrently. The driver switches between their CUDA contexts every few milliseconds, so each process sees itself running alone at roughly half speed. In the course lab this is measured directly: two subprocesses each running the same benchmark take almost exactly twice as long as one running alone.
The operational consequence is worth stating plainly. Schedule eight inference replicas onto one card by time-slicing and each takes eight times longer. You have the same queries per second you would have had with one replica, delivered at eight times the latency, and you have added the operational surface of eight pods. That is a loss dressed as consolidation.
In Kubernetes this mechanism appears as a device-plugin setting that advertises several replicas per physical GPU. The plugin is honest about what it does, and the word replicas invites people to read it as capacity. It is oversubscription, and it is genuinely useful for development and staging where several people want a card occasionally and nobody is measuring latency.
MPS: real concurrency, one shared context
The Multi-Process Service is a user-space daemon that funnels CUDA calls from many client processes through a single GPU context. With one context there is nothing to switch, so the context-switch overhead disappears and throughput with several clients approaches the sum of what they would each achieve alone, up to the point where the SMs saturate.
Two operational details matter. Every client process needs environment variables pointing at the daemon's pipes and logs, so joining MPS is something the workload opts into rather than something the platform imposes invisibly. And a per-client cap on the share of the SM grid is available, which is how you stop one greedy client starving the others.
What MPS does not give you is isolation. The clients share a context, so they share a fate: a fatal fault in one can take down the others, and nothing stops one client exhausting the memory the others were relying on. For cooperating workloads owned by one team this is usually acceptable. For two tenants who do not know each other, it is not.
On a cluster you would run the daemon per node, typically as a DaemonSet, rather than starting it by hand.
MIG: partition the card in hardware
Multi-Instance GPU splits an A100 or H100 class card into instances with their own streaming multiprocessors, cache paths and memory. The partition is enforced below the driver, which is what makes it the only option in this list with a real memory and fault boundary between tenants.
The trade is rigidity. Instances come from a catalogue of profiles rather than being sized freely, changing the geometry is a disruptive operation rather than a runtime decision, and a workload that needs a whole card cannot use a partitioned one. You are converting one flexible resource into several fixed ones, and you want to be reasonably confident about the shape of demand before you do it.
MIG is the answer when tenants must not be able to affect each other: separate teams, separate customers, anything with a regulatory story or a latency guarantee attached.
Choosing: what question are you answering
Three questions separate the four mechanisms cleanly.
Is the contention inside one process? Then it is streams, and the other three are irrelevant.
Do the workloads trust each other? If they are owned by one team and a crash in one is survivable for the others, MPS gives you the concurrency without the rigidity of a partition. If they do not trust each other, MPS is off the table whatever the throughput numbers say.
Do you need capacity or convenience? Time-slicing is the right answer surprisingly often, as long as everyone understands they are buying convenient access rather than throughput. It is the wrong answer whenever somebody is measuring latency.
The failure mode to avoid is configuring time-slicing because it is easy, then reporting the increased pod count as improved utilisation. Utilisation goes up on the dashboard because the card is busier switching contexts.
In Kubernetes, where this becomes a naming problem
All three of the cross-process mechanisms are configured through the device plugin rather than by hand, and the choice changes what the node advertises to the scheduler.
That is the part that bites. Under one MIG strategy the node advertises nvidia.com/gpu. Under another it advertises per-profile resources such as nvidia.com/mig-1g.10gb. A cluster where somebody changed the strategy and nobody updated the manifests produces workloads requesting a resource that nothing publishes, which reads like a capacity problem and is a naming problem.
The quieter version of the same failure is a card partitioned into seven instances while the plugin advertises one schedulable unit. Six instances sit idle, the scheduler reports the node full, and every layer looks healthy except the utilisation graph. If that sounds familiar, the symptom-by-symptom walk in GPU operator troubleshooting covers how to spot it.
The measurement that settles arguments
Whichever mechanism you choose, one benchmark tells you whether it worked: run the workload alone, then run N of them together, and compare per-workload wall-clock rather than aggregate utilisation.
If each instance takes roughly N times longer, you have time-slicing whatever the configuration claims. If each takes about as long as it did alone until the SMs saturate, you have real concurrency. Utilisation percentage cannot distinguish these two cases, which is why a fleet can report healthy numbers while delivering nothing extra.
That measurement is also the honest input to a cost conversation. "We consolidated four jobs onto one card and each still runs at full speed" is a claim somebody can check. "Utilisation went from fifteen to sixty percent" is not, because context switching raises it too.
Where to learn this properly
The distinctions above are easy to read and easy to forget, because they only become concrete once you have watched two processes halve each other's throughput on a card you are looking at. The AI Infrastructure Engineer course has you measure all four, write the device-plugin configuration for each Kubernetes strategy, and then use the result in a cost audit that ends in a specific recommendation.
For the full sequence this belongs to, see the AI infrastructure engineer roadmap, where sharing sits in the scheduling stage between placement and storage.