TL;DR: An LLM gateway is one endpoint that every app in your company calls instead of calling the model provider directly. It authenticates the caller, logs every request, prices every call, enforces per-app rate limits and daily budgets, caches exact repeats, retries transient failures with backoff, replays idempotent requests instead of paying twice, routes each request to a model by policy with a fallback, and produces the per-app report that explains the bill. Commercial gateways such as LiteLLM and Portkey package those capabilities with many providers and a dashboard. The core is a few hundred lines you can build in an afternoon, and building it once is the fastest way to evaluate the products.
Three internal apps at a company call the model provider with one shared API key. In September the bill doubles. The platform team asks which app did it, and the provider's invoice has one line. One app's retry loop had been re-sending the same failed request for hours, another had been sending its full system prompt with every keystroke of an autocomplete feature, and a third had hit the provider's rate limit hard enough to take the other two down with it. Nobody had a log. Nobody had a number per app. The fix was one box in the middle.
Build it, then judge the products
The LLM gateway lab in the AI Engineer course builds the gateway described below in eight steps, each checked against a deterministic fake upstream with a controllable clock, and finishes with a replayed morning of traffic and the per-app report. It is the hands-on version of the course's minimum-viable architecture lesson.
What an LLM gateway is
The demo architecture is a browser, an arrow and a provider. The first production architecture adds a backend between them, and everything an LLM feature needs to be operable lives in that backend: authentication, logging, rate limiting and observability. An LLM gateway is that backend factored out so that every app shares it. Apps call the gateway with their own key and an OpenAI-compatible request. The gateway holds the provider keys, applies policy, calls the provider, and returns the response with a few extra headers that say what it did.
The word "gateway" borrows from API gateways such as Kong and Envoy, and the resemblance is real. What is different about the LLM case is that every request has a price that depends on its size and its model, responses are slow enough that retries and timeouts dominate the error budget, and the same question arrives thousands of times a day in slightly different clothes.
The capabilities, and what each one prevents
Gateway capabilities, the failure each prevents, and the detail that makes it work
| Capability | What it prevents | The detail that matters |
|---|---|---|
| API keys and a request log | Unattributed spend; incidents with no evidence | Log the rejections too; put the request id in the response so users can quote it |
| Price per call | Reconstructing cost from the invoice a month later | Price the model the provider served, which can differ from the one requested; never price an unknown model at zero |
| Rate limits and daily budgets | One app taking the others down; a bug with no cost ceiling | Per key, decided before the upstream call; both answer 429 with retry-after and a type that says which limit fired |
| Exact-match response cache | Paying for the same answer hundreds of times | Key on the whole canonical request; a prefix key hands one user another user's answer |
| Retries with backoff | A provider hiccup becoming a user-visible error | Retry 429, 5xx and timeouts only; honour retry-after; add jitter; bound the attempts |
| Idempotency keys | A client retry charging twice for one answer | Store the response per key and replay it; scope keys per caller |
| Routing and fallback | Every app choosing its own model; an outage on one model failing all traffic | Resolve aliases by policy and tier; fall back on availability failures, never on a 400 |
| Per-app report | The bill that nobody can explain | Requests, blocked, errors, cache hit rate, spend and p95 latency per app, from the log |
Each row is a real gateway feature, and each one is small. What makes a gateway hard is the combination: the order in which the capabilities run, and the cases where a naive version of one quietly breaks another.
The order of operations
A request enters and the gateway decides, in this order: who is calling; which model this request resolves to; whether an idempotency key says the answer already exists; whether the cache has it; whether the caller is over a limit; then the call, with retries and a fallback; then pricing, spend accounting and the log line. In code the shape is a single method that delegates to one small function per stage.
def handle(self, req):
key = self.authenticate(req.headers) # 401 if unknown or revoked
route = self.route(req.body, key) # alias -> model id, fallback, or 403
if replay := self.idempotent_replay(req, key):
return replay
if hit := self.cache_get(self.cache_key(req.body, route.primary)):
return respond(hit, cache="HIT", cost=0)
if limited := self.check_limits(key): # 429 rate_limit or budget
return limited
status, data, attempts = self.call_with_retries(req.body, route.primary)
if status in RETRYABLE and route.fallback:
status, data, more = self.call_with_retries(req.body, route.fallback)
cost = self.price(data["model"], data["usage"])
self.record_spend(key, cost); self.cache_put(...); self.log(...)
return respond(data, cost=cost, served=data["model"])
The order encodes three decisions. Authentication comes first because nothing else can be attributed without it. The cache comes before the rate limiter because a cached answer costs nothing and should not spend a slot. The limiter comes before the upstream call because a limit applied after the money is spent is a report, and a limit applied before is a control.
Where gateways go wrong
The prefix cache. A cache keyed on the system prompt, or on the first few hundred characters of the request, looks like a higher hit rate. It is a data leak: two users with the same system prompt and different questions collide, and the second user gets the first user's answer. The safe cache is an exact match on everything that changes the output: the resolved model, every message, temperature, max tokens, tools and response format, hashed after canonical serialisation so key order and whitespace cannot split one request into two keys.
Retrying the wrong things. A 429 says slow down and usually says for how long. A 5xx is the provider having a bad moment. A timeout means you do not know what happened. All three deserve a bounded retry with a doubling delay and a random jitter, so a thousand clients do not retry in synchronised waves. A 400 says the request is malformed, and it will be malformed on the next attempt and on the fallback model too. Retrying it spends money to see the same error, and falling back on it hides the real problem from the caller.
Pricing the requested model. Providers and proxies report the model they served in the response, and the name can differ from the one in the request. A pricing table keyed on the requested name silently prices some traffic at the wrong rate, or at a default rate, or at zero. The gateway should price the served name and treat a model with no price entry as expensive, so it shows up in the report as a spike instead of as free.
Limits without identity. A rate limit applied to all traffic is a limit on the company. A budget without a key is a budget for everyone. The per-app report is only possible because every request carries a key that maps to a user and an app, and the rejections are logged as carefully as the successes.
Retries at every layer. When each app retries on its own and the gateway retries as well, one bad minute at the provider becomes a storm. The gateway should own the retry policy, and apps that must retry after a network error should send an idempotency key so the gateway replays the stored response instead of calling the provider again.
LLM router or LLM gateway
The terms overlap. An LLM router is the routing capability on its own: given a request and a policy, pick the model. Routing by prompt size and tool use, routing by tier, cascading from a small model to a large one when the small one is unsure, and falling back when a model is down are all routing. A gateway is the router plus everything else on the table above. Products marketed as routers usually grow the rest of the table within a year, because the routing decision needs the identity, the log and the prices to be worth anything.
Build or buy
Build the core once. A gateway with the capabilities above, against one provider, is a single class behind a plain HTTP server and fits in a few hundred lines. Building it teaches the order of operations and the failure cases, and it gives you a working reference to compare products against. Many teams run that small gateway in production for a long time, because a single provider and a handful of apps do not need more.
Buy, or adopt an open-source gateway, when the list of providers grows past two, when you need a dashboard that non-engineers read, when guardrails and PII masking belong at the gateway, or when semantic caching is worth its false-positive risk for your traffic. LiteLLM's proxy documents virtual keys, budgets, rate limits, fallbacks and spend tracking per key and team, and Portkey documents caching, retries, fallbacks and load balancing in a single configuration. When you evaluate either, the table above is the checklist, and the questions from the previous section are the interview: how is the cache keyed, what is retried, which model name is priced, and what exactly is logged for a rejected request.
The production-serving side of the course covers the cascade that sits inside the routing stage, and the observability guide covers the tracing that a gateway's log feeds.