Live suggestions for a telecom contact center

Lumora Telecom (fictional), a mobile operator, wants live suggestions for its 6,000 contact-center agents. While a call runs, the system sends the transcript so far to a language model every 15 seconds and shows the agent a suggested next reply. When the call ends, it writes a summary into the CRM.

Assume:

• Model: 32 billion parameters, dense, 64 layers, grouped-query attention with 8 KV heads of dimension 128. Weights and KV cache in FP8 (1 byte per value).
• Busiest hour: 70% of agents are on a call. An average call lasts 6 minutes.
• A suggestion request holds:
  • a system prompt of 1,200 tokens, the same for every call;
  • retrieved knowledge snippets, 900 tokens, different on every request;
  • the transcript so far, 600 tokens on average, growing by about 50 tokens between requests;
  • an instruction of 100 tokens.
  • The output is 80 tokens.
• The customer's developers build the prompt in this order: system prompt, snippets, transcript, instruction.
• A summary request at the end of each call: 2,500 tokens in (system prompt, full transcript of 1,200 tokens, instruction) and 250 tokens out, due within 60 seconds.
• Suggestion SLOs: time to first token (TTFT) under 600 ms at p95 and time between tokens (TBT) under 40 ms at p95. A suggestion more than 10 seconds late is useless.
• GPUs: H100 SXM, 8 per node, NVLink inside the node. Per GPU: 1,979 TFLOPS of dense FP8 compute, 80 GB of memory at 3.35 TB/s. Plan on 40% of peak FLOPS in prefill and 70% of peak memory bandwidth in decode.
• The customer owns 4 such nodes. Another node costs $280,000 installed.

Constraints:

• Call audio and transcripts must stay in the country, on the customer's premises.
• This year's hardware budget is $600,000.
• The ops team runs Kubernetes and has never run LLM serving.

Asks:

1. How many GPUs do they need at the busiest hour? Split prefill and decode. Are their 32 GPUs enough?
2. Design the serving system: the product layer (NIM, vLLM, TensorRT-LLM or Triton), batching, caching, routing and scaling.
3. A whole node fails at the busiest hour. What do agents see, and what do you build so that it is acceptable?
4. How do you roll out a new model version and show that it is better?

Write down the requests per second, the uncached prompt tokens per request, and your GPU count split into prefill and decode. Then answer the customer's real question in one line: are their 32 GPUs enough?