Step 1: In-process sharing: CUDA streams
A CUDA stream is an ordered queue of kernel launches and memcpys. Two kernels on two different streams may execute concurrently on the GPU if the hardware has SMs to spare. Two kernels on the same stream run strictly in order.
The default stream (stream 0) serializes everything. That's fine for simple scripts, but inside a serving process that handles multiple simultaneous requests you want overlap — image preprocessing on stream A while the previous batch's forward pass completes on stream B.
Why can't two kernels always overlap? A single large GEMM uses 100% of SMs for its duration — there's no room. Overlap only happens when at least one kernel leaves SMs idle (small batch, memory-bound, or launch-bound). That's the common case for serving: lots of small kernels.
Below: launch two independent 1024×1024 GEMM chains on two streams; compare to running them back-to-back on one stream.
Setup — two independent workloads
Two GEMM chains that share no data. Because there's no data dependency between them, the CUDA runtime is free to overlap their kernels if we give it the hint (a separate stream per chain). Warm-up is important — the first few launches pay JIT and cache setup costs that would otherwise dominate the measurement.
import torch, time
device = 'cuda'
N = 1024
ITERS = 200
def gemm_chain(x, y, iters=ITERS):
for _ in range(iters):
x = x @ y
return x
# Two independent workloads that don't share data — nothing forces ordering.
a1 = torch.randn(N, N, device=device)
a2 = torch.randn(N, N, device=device)
b1 = torch.randn(N, N, device=device)
b2 = torch.randn(N, N, device=device)
# Warm up CUDA kernels so the measurement isn't dominated by first-launch JIT
for _ in range(3):
gemm_chain(a1, a2, iters=10)
gemm_chain(b1, b2, iters=10)
torch.cuda.synchronize()
Serial baseline — both chains on the default stream
By default, every CUDA call goes on the default stream, and the default stream implicitly synchronizes with every other stream. So two back-to-back chain calls run strictly one after the other — no overlap possible.
# --- Serial: both chains on the default stream, back-to-back ---
torch.cuda.synchronize()
t0 = time.perf_counter()
gemm_chain(a1, a2)
gemm_chain(b1, b2)
torch.cuda.synchronize()
serial_ms = (time.perf_counter() - t0) * 1000
Concurrent — one stream per chain
Putting each chain on its own torch.cuda.Stream tells the runtime: these have no ordering dependency; overlap them if the hardware has capacity. A single 1024×1024 GEMM doesn't saturate all the SMs on a modern GPU, so the second stream can use the spare SMs — giving a real speedup. At larger matrix sizes that do saturate the SMs, the streams converge back to serial.
# --- Concurrent: one stream per chain ---
s1, s2 = torch.cuda.Stream(), torch.cuda.Stream()
torch.cuda.synchronize()
t0 = time.perf_counter()
with torch.cuda.stream(s1):
gemm_chain(a1, a2)
with torch.cuda.stream(s2):
gemm_chain(b1, b2)
torch.cuda.synchronize()
streams_ms = (time.perf_counter() - t0) * 1000
streams_result = {
'serial_ms': serial_ms,
'streams_ms': streams_ms,
'speedup': serial_ms / streams_ms,
}
print(f"Serial (1 stream): {serial_ms:7.1f} ms (both chains back-to-back)")
print(f"Concurrent (2 streams): {streams_ms:5.1f} ms (overlapped)")
print(f"Speedup: {streams_result['speedup']:6.2f}x")
print()
print("Interpretation: a single 1024x1024 GEMM leaves some SMs idle on this GPU,")
print("so two concurrent chains exploit the spare capacity. With larger matrices")
print("that fully saturate SMs, the streams version converges to serial.")
