Review a teammate's design doc for a streaming API

You are on the inference team at Larkspur AI (fictional). A teammate wrote the design below for a new API. Launch is in three weeks. Review it:

1. Present your top 3 problems, each with evidence and a fix.
2. List everything else that is wrong, risky or missing.
3. Defend your findings when the interviewer pushes back.

Assume an H100 gives 989 TFLOPS dense BF16 and 3.35 TB/s.

Design: Brief API v1 (streaming summaries for Corvel Legal)

Goal. Corvel Legal (fictional) sells contract software to law firms. Its users paste a contract and get a streamed summary and a list of risky clauses. We serve this with our 27B model.

Model. 27B parameters, dense, BF16. 48 layers, hidden size 4,608, 16 KV heads of dimension 128. Context up to 64k tokens.

Traffic. Peak is 50 requests a second during US and EU business hours. Prompts average 2,600 tokens and outputs average 400 tokens. About 3% of prompts are over 32k tokens.

Hardware. H100 80 GB GPUs. Each replica is one GPU.

Memory. The weights take 54 GB, which leaves about 20 GB for the KV cache after overhead. The KV cache needs layers × KV heads × head dim × 2 bytes = 48 × 16 × 128 × 2 = 192 KiB per token. 20 GB holds about 100,000 tokens: 32 average requests per replica, and a 64k request fits.

Batching. The scheduler collects requests for 250 ms, then runs them as one batch (up to 32) until every request in it has finished. Requests that arrive meanwhile wait for the next batch.

Capacity. We load-tested one replica with 8 client threads sending 1,000 copies of one 2,000-token prompt, each asking for 128 tokens. It sustained 2.2 requests a second at 85% GPU utilization. Peak is 50 requests a second, so we need 50 / 2.2 = 23 replicas. We'll run 30, 10 in each of 3 availability zones, for 30% headroom.

Scaling. Add replicas when average GPU utilization is above 85% for 10 minutes. A new replica takes about 7 minutes to start (pull the 18 GB image, download 54 GB of weights from object storage, load).

Deploys. New model versions go out by restarting all replicas at 02:00 UTC, our quietest hour.

Clients. The SDK retries a request up to 3 times if it gets no response within 30 s.

SLO. p50 end-to-end latency under 2 s.

Health. A Kubernetes liveness probe calls /health, which returns 200 when the server process is up.

Read the doc in 10 minutes. Write your top 3 problems. For each, quote the number or sentence in the doc that proves it, and write a fix.