Inference Serving Patterns: Dynamic Batching, Throughput, and the Triton Mental Model

You have a model. Now a thousand users want to query it simultaneously. Your first instinct — wrap it in Flask/FastAPI, one request at a time — wastes 90% of your GPU. A modern inference server like NVIDIA Triton Inference Server turns a bare model into a production-grade endpoint with dynamic batching, multi-model sharing, versioning, and built-in Prometheus metrics.

This lab teaches the mental model of inference serving by building a mini-Triton in Python. Why build it instead of just running Triton? Because once you've implemented dynamic batching in 30 lines, you understand why Triton's config.pbtxt has the knobs it has — and you can debug your production Triton deployment when it misbehaves.

The core insight: GPUs are batch-mode accelerators

A GPU doing matrix multiplication on batch=1 runs at maybe 5% of its peak FLOPS. The tensor cores need work — large matrices — to be saturated. Serving requests one at a time is like hiring a 1,000-person factory and giving them one order every minute.

Dynamic batching is the fix: incoming requests wait in a queue for up to N milliseconds OR until the queue has B requests, whichever comes first. Then the server fires a single large batched forward pass. Each individual request waits a bit longer, but aggregate throughput goes 5-50×.

References

• NVIDIA Triton Inference Server — the real thing. The concepts in this lab map 1:1 to Triton's config.pbtxt. Deployment = one Docker container.
• Triton model configuration docs — every knob explained. Read after this lab.
• Serving ML Models — Chip Huyen — excellent breakdown of online vs batch vs stream serving.
• Continuous batching (Anyscale) — LLM-specific evolution of dynamic batching (used by vLLM, TGI, Triton's vLLM backend).
• Orca: A Distributed Serving System for Transformer-Based Generative Models — OSDI 2022, the paper behind iteration-level scheduling / continuous batching.

---

A ResNet-style convnet classifying 224×224 images. We'll simulate concurrent client load and measure QPS (queries per second) and latency percentiles.

Expected result on an RTX 3060 Ti: ~20-40 QPS, which is terrible — the GPU sits mostly idle between requests. The p99 latency will be close to the mean because there's no queue contention, but throughput is severely capped.

import torch, torch.nn as nn
import time, queue, threading

device = torch.device('cuda')
torch.manual_seed(0)

# A realistic-size image model (~12M params)
model = nn.Sequential(
    nn.Conv2d(3, 64, 7, stride=2, padding=3), nn.ReLU(),
    nn.Conv2d(64, 128, 3, stride=2, padding=1), nn.ReLU(),
    nn.Conv2d(128, 256, 3, stride=2, padding=1), nn.ReLU(),
    nn.Conv2d(256, 512, 3, stride=2, padding=1), nn.ReLU(),
    nn.AdaptiveAvgPool2d(1), nn.Flatten(),
    nn.Linear(512, 1000),
).to(device).half().eval()

# Warmup
with torch.no_grad():
    for _ in range(3):
        _ = model(torch.randn(1, 3, 224, 224, device=device, dtype=torch.float16))
    torch.cuda.synchronize()

@torch.no_grad()
def infer_naive(x):
    """One request, one forward pass. Classic REST-endpoint pattern."""
    return model(x)

print('Model ready. ~12M params, fp16.')

The load-test harness

A thread-per-client test rig that submits n_requests through handler_fn using concurrency parallel threads. Measures QPS and latency percentiles — the two numbers every inference SLO is written against. torch.cuda.synchronize() after each call makes the timer measure actual GPU execution, not just CUDA API submission.

# Simulate N concurrent clients each firing requests
def run_load_test(handler_fn, n_requests=200, concurrency=16):
    """Fires n_requests through handler_fn using `concurrency` threads.
    Returns QPS + latency percentiles."""
    req_queue = queue.Queue()
    for i in range(n_requests):
        req_queue.put(i)
    latencies = []
    lock = threading.Lock()

    def client():
        while True:
            try:
                req_queue.get_nowait()
            except queue.Empty:
                return
            x = torch.randn(1, 3, 224, 224, device=device, dtype=torch.float16)
            t0 = time.perf_counter()
            _ = handler_fn(x)
            torch.cuda.synchronize()
            with lock:
                latencies.append((time.perf_counter() - t0) * 1000)

    threads = [threading.Thread(target=client) for _ in range(concurrency)]
    start = time.perf_counter()
    for t in threads: t.start()
    for t in threads: t.join()
    elapsed = time.perf_counter() - start

    qps = n_requests / elapsed
    latencies.sort()
    p50 = latencies[len(latencies) // 2]
    p99 = latencies[int(len(latencies) * 0.99)]
    return qps, p50, p99

Baseline — naive one-at-a-time serving

Fire 200 requests through the naive handler at concurrency=16. Each request pays the full kernel-launch + forward-pass cost on its own. You'll see QPS bottlenecked well below the GPU's throughput capacity — and p99 latency dominated by CUDA stream contention.

print('Running naive-serving load test (200 requests, 16 clients)...')
naive_qps, naive_p50_ms, naive_p99_ms = run_load_test(infer_naive)
print(f'  QPS : {naive_qps:.1f}')
print(f'  p50 : {naive_p50_ms:.1f} ms')
print(f'  p99 : {naive_p99_ms:.1f} ms')