The booking assistant got slow, and nobody deployed anything

Skyrail Travel (fictional), an online travel agency, runs a booking-change assistant in its web chat and app. The customer's engineering lead opens the call: "Over about two weeks, the p95 time for an assistant turn went from 6.8 seconds to 18.5 seconds. Customers are abandoning chats. Nothing was deployed. Help us find it."

The system, as they draw it:

• Chat UI, then an API gateway, then an orchestrator.
• The orchestrator calls a router (a small model) that classifies the request.
• Then an agent (a large model) runs a tool loop with up to 3 model calls per turn. Its tools are get_booking, search_fares, change_booking and policy_lookup.
• The final reply streams to the customer.
• About 64,000 turns a day.

Assume:

• Large model: $3 per million input tokens, $15 per million output tokens. Small model: $1 and $5.
• The provider caches prompt prefixes. A request reuses the cache only if its prompt starts with exactly the same tokens up to the cache breakpoint. Reading cached tokens costs 10% of the input price. Writing a new cache entry costs 125% of the input price.
• Time to first token is about 0.3 s plus the uncached input tokens divided by 8,000 tokens a second. Cached tokens add little time.
• Output streams at about 100 tokens a second.

Your tasks:

1. Rank your hypotheses and say what evidence you want first.
2. Find the root cause with the evidence you are given.
3. Fix it, and show that the fix explains the numbers.
4. Make sure it cannot happen again.

Write your first five hypotheses, ranked by how likely they are and how cheap they are to check, and the first piece of evidence you will ask for. The interviewer holds the evidence and hands over only what you ask for.