Serve a 70B coding assistant at 400 requests a second

You run serving for a coding assistant built on a 70B model. Users see the reply stream into their editor.

The model:

• 70 billion parameters, weights in BF16.
• 80 layers, hidden size 8,192, 64 query heads.
• Grouped-query attention with 8 key-value heads of dimension 128.

Traffic:

• Peak traffic is 400 requests a second.
• The average prompt is 3,000 tokens. 60% of requests share the same 2,000-token system prefix.
• The average output is 250 tokens.
• Free-tier users send 30% of requests. Under overload they may be slowed or turned away. Paid users may not.

SLOs:

• Time to first token under 800 ms at p95.
• Time between tokens under 60 ms at p95.

Hardware and price (assumptions):

• 8-GPU H100 nodes. Per GPU: 80 GB of memory, 3.35 TB/s of memory bandwidth, 989 TFLOPS dense BF16, NVLink at 900 GB/s inside the node.
• An H100 costs $2.50 an hour, all-in.

Asks:

1. How many GPUs do you need at peak? Split prefill and decode.
2. Design the serving system: batching, caching, routing, autoscaling.
3. A GPU dies mid-generation. What happens to the requests on it?
4. How do you roll out a new model version?

Write down three things in 10 minutes: the memory one replica needs (weights plus cache), your GPU count for prefill and for decode at peak, and your top three risks.