Build an LLM Gateway: One Door Between Your Apps and the Model
Put a gateway you wrote in front of the model and give it the four floor capabilities every LLM feature needs: API keys with a request log, a price on every call, per-app rate limits and daily budgets, an exact-match response cache, retries with backoff and idempotency keys, model routing with a fallback, and a report over a replayed morning of traffic that shows who spent what.
Hands-on labs require Pro · $29.99/mo · cancel anytime
What you'll learn
- 1Pass-through: one door, and the model behind itNorthwind's three apps call the model with one shared provider key, from
- 2Keys and the log: who is calling, and a record of every callThe architecture lecture's floor has four capabilities, and two of them are
- 3Price every call on the model that was actually servedThe bill is the sum of every call's tokens times the model's rate. The
- 4Rate limits and daily budgets: stop the spend before it happensThe provider has rate limits, but they protect the provider. When the
- 5An exact-match cache that cannot collideThe support bot answers "How do I reset my password?" a few hundred times a
- 6Retries with backoff, and idempotency keys so clients can retry tooProviders fail in three ways that deserve a retry and one that does not.
- 7Route by prompt and tier, fall back when a model is downApps have been sending fast and smart since Step 1, and the gateway has
- 8Replay a morning of traffic and report who spent whattraffic.jsonl is a recorded morning: the support bot asking its usual
Prerequisites
- Python: classes, dicts, json, try/except, a while loop
- HTTP basics: status codes, headers, JSON bodies
- Read the module's caching, rate-limit and architecture lectures first
Exam domains covered
Skills & technologies you'll practice
This intermediate-level ai/ml lab gives you real-world reps across:
Why put a gateway between your apps and the model
The demo architecture is a browser, an arrow and a provider. The first production architecture adds one box in the middle, and that box is where authentication, logging, rate limiting and observability live. Commercial gateways sell those four things plus caching, retries and routing. Building a small one yourself, on the plain chat completions API, shows what each of those features costs to get right and makes the commercial ones easy to evaluate. The lab is a single Python class behind a plain HTTP server. Each step adds one capability and tests it against a deterministic fake upstream: keys that are attributed and revoked, a request log, prices keyed on the model the provider actually served, a sliding rate-limit window and a daily budget, an exact-match cache that must not collide across users, exponential backoff that honours retry-after and stops for non-retryable errors, idempotency keys that make client retries safe, routing by prompt size and tier with a fallback model, and a per-app report over a replayed morning of traffic with cache hit rates, spend and p95 latency.