Score card payments inside a 40 ms budget

Tallyline is a card processor (fictional). Every card authorization goes through a fraud model before the bank approves or declines it. You own the scoring service. Treat the model as a black box that runs on GPUs.

Traffic:

• Live authorizations: 6,000 a second on a normal afternoon, 24,000 a second at the yearly peak.
• Retro-scoring: the disputes team rescores old transactions at a steady 3,000 a second. Each result is needed within 5 seconds.

Latency:

• The payment gateway gives the fraud check 40 ms at p99, measured at the gateway. After that it falls back to a rules engine, which the risk team says catches less fraud.
• Network round trip between the gateway and the scoring service: 3 ms.
• Feature lookup (the card's last 64 transactions) from a feature store: 7 ms at p99. It must finish before the model runs.
• CPU pre- and post-processing: 2 ms.

The model, as measured by the model team (assumptions):

• One GPU runs one batch at a time. A batch of b inputs takes t(b) = 6 ms + 0.4 ms × b.
• Memory caps a batch at 128 inputs.
• Hosts have 8 GPUs. A GPU costs $2.00 an hour.
• The service runs in 3 availability zones. Losing one zone must not stop live scoring.

Asks:

1. Work out the time left for queueing and the GPU. What batch size and dispatch rule do you use, and how many authorizations a second can one GPU take inside the budget?
2. How many GPUs do you need for the peak, including the zone rule?
3. Design the service: intake, batching, routing to GPUs, returning each score to the right caller, the fallback, and the retro-scoring stream.
4. A GPU hangs in the middle of a batch of 30 live authorizations. What happens to them?

Write the latency budget as a sum (network, features, CPU, queue, GPU) and the largest batch it allows. Then write your rule for when a batch goes to the GPU, in one sentence.