Streaming and Prompt Caching: Time to First Token, SSE and Cheaper Prompts

Stream LLM responses and measure time to first token, parse Server-Sent Events by hand, detect a buffering gateway, reorder a prompt so the provider's prefix cache serves 97 % of it, cancel generation when the user stops, and relay a stream through your own FastAPI backend without buffering.

50 min5 steps3 domainsIntermediate

Hands-on labs require Pro · $29.99/mo · cancel anytime

What you'll learn

1
Stream and time it
Pennant Outdoor's support assistant answers in about 120 words. Without streaming, the customer stares at a
2
Read Server-Sent Events yourself
Under the SDK, a stream is plain text over HTTP. Each event is a line starting with data: followed by a blank
3
Prompts that hit the cache
Every request carries the same 3,000-token policy. Providers cache prompt prefixes: when a request starts
4
Stop when the user does
Customers press Stop, close the tab or send a new question. If your code just stops showing the answer but keeps
5
Relay without buffering
Step 1 showed a gateway that held the whole answer back. Your own backend sits on the same path: the browser

Step 1, as you will see it

This is the lab’s own text. Each step ends with a check that runs your work in the lab environment; the hint and the solution stay inside the lab.

Step 1: Stream and time it

Pennant Outdoor's support assistant answers in about 120 words. Without streaming, the customer stares at a spinner until the last word is generated. With streaming, words appear as they are produced, so the wait that matters is time to first token (TTFT).

llmsim.py is a small model server that behaves like a hosted provider: it streams Server-Sent Events, its TTFT grows with the prompt it has to read, and it caches prompt prefixes. The Run button starts it on port 8090. Its answers are filler text; what matters here is how they arrive.

Do this

1. Write stream_chat(client, messages, max_tokens): call client.chat.completions.create(...) with stream=True and stream_options={"include_usage": True}, loop over the chunks, and return {"text", "ttft", "total", "chunks", "usage"}: the joined text, seconds to the first non-empty delta.content, seconds to the end, the number of text pieces, and the final chunk.usage.model_dump().

2. Write is_buffered(result): True when the first text arrived at 80 % or more of the total time.

3. Run. It streams from the simulator and from the lab's real model gateway. Compare the two lines.

Prerequisites

Python: functions, loops, generators (helpful)
Helpful: Tokens, Context and Cost

Exam domains covered

LLM applicationsLatencyCost

Skills & technologies you'll practice

This intermediate-level ai/ml lab gives you real-world reps across:

streamingprompt cachingSSEtime to first tokenFastAPIintermediate

LLM streaming and prompt caching, measured

Streaming changes what users wait for: the time to the first token instead of the whole answer. Prompt caching changes what you pay for: a repeated prefix is read once and then served cheaper and faster. Both are easy to lose, to a buffering proxy, a relay that collects before it sends, or a timestamp at the top of the prompt. In this lab you measure time to first token, parse Server-Sent Events yourself, detect buffering, reorder a prompt for the prefix cache, cancel generation early and relay a stream through FastAPI, against a local model server that behaves like a hosted provider.

Frequently asked questions

What is time to first token?

The time from sending a request to receiving the first piece of the answer. With streaming it is what users perceive as the wait; the rest of the answer arrives while they read.

How does prompt caching work?

Providers cache the beginning of recent prompts. When a new request starts with exactly the same tokens (often at least 1,024 of them), that prefix is served from the cache at a lower price and with less delay. Anything that changes, such as a timestamp or a user name, must come after the stable part.

Why is my LLM stream arriving all at once?

Something on the path is buffering: a proxy, a gateway or your own backend collecting the answer before sending it. The lab detects it by comparing time to first token with total time.

Does stopping a stream stop the cost?

Only if you close the connection. If your code stops displaying the answer but keeps reading, the model keeps generating and you pay for every token.