Live call assist on four GPUs

Brenmoor Energy, a fictional national energy supplier, runs a call center that is open from 07:00 to 21:00. During each call, a speech-to-text system turns the customer's words into text. When the customer finishes an utterance, an assist service suggests the agent's next best action on screen: one of 140 action codes, such as "offer a payment plan" or "book an engineer visit".

The service runs a fine-tuned open-weights model on the customer's own GPUs:

• The model has 1.5 billion parameters and runs in BF16. It has a classification head, so each request is one forward pass with no token-by-token generation.
• Each request carries about 384 tokens: the last two turns plus a short instruction block.
• The customer's benchmark: a batch of b requests takes t(b) = 10 ms + 2.5 ms x b on one GPU. GPU memory allows batches of up to 64.
• Assume each GPU peaks at 989 TFLOPS of dense BF16 compute.

Traffic:

• Normal peak: 3,500 calls at once. Storm or outage peak: 10,500 calls at once.
• A customer finishes an utterance about every 7 seconds.
• About 35% of utterances are under four words ("yes", "okay, thanks").
• The center takes about 5.5 million calls a month, averaging 7 minutes each.

Constraints:

• The SLO: p99 under 180 ms from the moment the service receives a request to the moment it answers. Tokenizing, the network inside the data center and post-processing take 12 ms of that.
• The pool is 4 GPUs in one server. It is fixed for this quarter: procurement takes 10 weeks, and storm season starts in 3 weeks.
• Transcripts may leave the customer's data centers only through a provider that security has approved. Approval takes about 8 weeks.

The pilot ran with 280 agents. Its batcher kept the settings from the serving framework's example: send a batch when it has 48 requests, or 200 ms after the first request in the batch arrived. Requests are spread round-robin across the 4 GPUs. The pilot measured a p50 of 160 ms and a p99 of 232 ms, so it misses the SLO. The GPU logs show batches of 2 to 4 requests that take 15 to 20 ms each, and each GPU is busy about 6% of the time. The operations lead asks: "The GPUs sit idle most of the time, and agents still get their suggestions late. Where does the time go?"

Prices for task 4:

• Owned GPUs cost $2.20 per GPU-hour all-in (depreciation, power, space and a share of operations).
• A hosted small model costs $1 per million input tokens and $5 per million output tokens, and cached input costs 10% of the input price. Without fine-tuning, the hosted prompt needs about 2,400 tokens of instructions and examples (the same on every request), plus about 120 tokens of conversation and about 8 output tokens.

Your tasks:

1. Explain the pilot's latency with numbers, and fix it.
2. How many requests a second can one GPU serve? How many GPUs do normal peak and storm peak need within the SLO, including the loss of one GPU?
3. Design the batching, routing, admission and degradation so storm season works on 4 GPUs.
4. The CFO asks whether to drop the GPUs next year and call a hosted API instead. Compare the monthly cost and name the other factors.

Write down the throughput of one GPU at three batch sizes, the number of GPUs you need at storm peak, and your one-sentence explanation of why the pilot misses its SLO.