TL;DR: LLM observability is the ability to answer, for any request your assistant handled, four questions: how long it took and where the time went, what it cost and why, what the model was given and what it produced, and whether the answer was any good. The mechanism is a trace, one per request, made of nested spans with timing, status and attributes. Everything else in the field is built on that: percentiles per stage, cost per answer computed from the API's own token counts, sampling that keeps every failure and a slice of the rest, redaction before anything reaches disk, and a gate that compares a prompt change against a baseline before it ships. This guide walks through each piece with the code shape, and the paired lab has you build the whole layer by hand around a working RAG assistant.
A support assistant goes live on a Tuesday. On Friday finance asks why the model bill is three times the estimate. The following week a customer forwards a confidently wrong answer to their account manager. Both questions land on the same engineer, and the honest answer to both is the same: the service worked, and nobody can say what it did. The logs show requests arriving and responses leaving. They do not show which stage took the time, which requests burned the tokens, or which retrieval returned the wrong document.
That gap is what LLM observability closes. The term gets used for dashboards, for vendor products and for evaluation suites, and all of those are downstream of one thing: a trace of each request, recorded at the source with enough structure to be queried. Get the trace right and the dashboards are a query. Get it wrong and every incident starts with reading transcripts.
Build it, then read it
The LLM observability lab puts a working RAG assistant in a sandbox and has you write the tracer, the cost model, the report, the sampler, the redaction and the regression gate yourself, step by step, with a checker that runs your code. It sits in the evaluation and MLOps module of the AI Engineer course. This article is the reading that goes with it.
Why an LLM app needs its own observability
Conventional application monitoring assumes that a request either succeeds or throws, that a successful response is a correct one, and that cost is roughly proportional to request count. An LLM application breaks all three assumptions at once.
A request can complete with status 200 and a fluent, wrong answer, because the retrieval step handed the model the wrong document and the model did what models do with whatever context they are given. Nothing raised. Cost varies per request by an order of magnitude, since it is a function of prompt length and completion length rather than of the request happening at all. And latency is dominated by a single external call whose duration depends on output length, so the median tells you little about what one customer in twenty experiences.
The consequence is that the unit of observation has to be the request as a tree of stages, with the model's own accounting attached, and the quality signal has to be recorded alongside timing rather than bolted on later. That is a trace.
The trace: one request, a tree of spans
A trace is one request end to end. A span is one timed unit of work inside it, and spans nest. For a retrieval-augmented assistant the tree is short: a root span for the whole answer, a child for retrieval, a child for generation. Each span carries six structural fields and a bag of attributes.
{
"trace_id": "9c453820e2b04f0f8b7d3a1c5e6f7a8b",
"span_id": "5d2a9e1c7b3f4a60",
"parent_span_id": "a1f0c3d4e5b6a798",
"name": "generate",
"start_ns": 1758124533211000000,
"end_ns": 1758124535721000000,
"duration_ms": 2510.6,
"status": "ok",
"error": null,
"attributes": {
"model": "meta-llama/llama-3.1-8b-instruct",
"prompt_tokens": 442,
"completion_tokens": 32,
"cost_usd": 0.0000948
}
}
The parent link is the field that turns a log line into a trace. With it, a query engine can rebuild the tree, attribute a slow root to the child that caused it, and join the retrieval scores of a request to the answer it produced. Every tracing standard exposes these fields under slightly different names; the OpenTelemetry semantic conventions for generative AI define a shared vocabulary for the model, token usage and related attributes, and the hosted products map onto it.
What goes into the attributes depends on the span:
What each span records
| Span | Attributes worth recording | Question it answers |
|---|---|---|
| answer (root) | input, output, total tokens, total cost, sources used | What did this customer ask, what did we say, what did it cost |
| retrieve | query, top-k, sources, scores, best score | Did we find the right document, and how confident was the search |
| generate | model, prompt variant, prompt tokens, completion tokens, cost, finish reason | Which prompt and model produced this, and what did the call consume |
| tool call (agents) | tool name, arguments, result size, status | Which tools ran, in what order, and which one failed |
Two rules make the attributes useful later. Record the model name the API returned rather than the one you asked for, because a gateway may serve a differently named deployment and that name is what gets billed. And record the retrieval scores on every request, since they are the cheapest quality signal you will ever get and they cost nothing to keep.
The signals worth a dashboard
Individual traces answer questions about one request. Operations questions are about populations, and a small set of aggregates covers most of them.
Latency percentiles per span. The p50 is the typical experience. The p95 is what one customer in twenty lives with, and it is the number that pages a service. Compute both per span name, and report each span's share of the root's time. In a typical RAG assistant the generation call owns most of the root and retrieval a thin slice, which settles the "make it faster" conversation before anyone guesses. Use the nearest-rank method so the p95 is always a latency that actually occurred and can be looked up.
Tokens and cost per answer. The mean cost per answer is the unit economics of the feature. Its distribution tells you whether a few long conversations are carrying the bill.
Retrieval quality. The best cosine score of each retrieval, tracked over time, exposes both drift in the corpus and the questions your knowledge base cannot answer. The next section on triage shows how to use it.
Errors by stage. Upstream timeouts, rate limits, malformed tool arguments and refusals all look different in a trace and identical in a status code.
An outcome signal. Thumbs up or down, an escalation to a human, a follow-up question on the same topic. Any of these joined to the trace id turns the trace store into an evaluation set.
Cost per answer, computed at the source
Every chat completion response carries a usage block with the real prompt and completion token counts. Read it from the response and set it on the generation span. Never estimate tokens from string length when the provider reports the actual count, and never rely on the monthly invoice as the only cost signal.
Pricing is a lookup on the model name with input and output priced separately, since output tokens typically cost more:
def cost_of(model, prompt_tokens, completion_tokens, pricing):
rate = pricing["models"].get(model) or pricing["models"].get("default")
if rate is None:
raise KeyError(model)
return round(prompt_tokens / 1e6 * rate["input_per_1m"]
+ completion_tokens / 1e6 * rate["output_per_1m"], 8)
The default entry matters more than it looks. When a new model name appears, from a gateway rename or a fallback route, an unknown model must cost something conservative rather than zero, or the report will quietly show a saving that is really a blind spot.
With cost on every span, roll it up to the root, and you have three things at once: a per-request figure for the customer who asks, a per-feature figure for the product manager, and the input to a budget alarm that fires before the invoice does.
Finding the wrong answer from the trace, before the customer does
Most wrong answers from a RAG assistant share an anatomy. The question is outside the knowledge base, the vector search returns the nearest chunks anyway because it always returns something, and the model answers from that noise with full confidence. Reading transcripts to catch this does not scale. The trace already holds the evidence: the retrieval span's best score sits well below the scores of questions the corpus covers.
A triage job over a day of traces is then a filter: every root whose retrieval score fell below a threshold, sorted worst first, with the question and the documents that were returned. That list is a prioritised backlog of documents to write. The threshold is a judgement that moves as the corpus and the embedding model change, which is why it belongs in a named constant that is printed with every run. Absolute scores depend on the embedding model. The gap between the in-scope cluster and the misses is what you are looking for.
Keep every bad request, sample the rest
At six requests a run, keep everything. At a million requests a day, recording every span of every request costs more in storage and query time than it returns, and the ordinary requests all look alike. The traces that carry information are the failures, the slow ones and the expensive ones.
The decision belongs at the end of the trace, when status, duration and cost are known:
def keep(root):
if root.status == "error":
return True
if root.duration_ms > slow_ms:
return True
if root.attributes.get("total_cost_usd", 0) > cost_alert_usd:
return True
bucket = int(hashlib.sha256(root.trace_id.encode()).hexdigest()[:8], 16) / 0xFFFFFFFF
return bucket < sample_rate
Two details are deliberate. The always-keep rules run before the sample, so the sampler can never drop the request you will be asked about. And the sample is a hash of the trace id rather than a random draw, so the same trace gets the same decision on every replay. A bug report that reproduces on Monday and disappears on Tuesday because the sampler rolled differently is the kind of thing that makes teams turn sampling off entirely.
Head sampling with an always-keep tail is where every production tracing pipeline converges. The report over the kept ordinary traces remains statistically honest, and the store stays small.
Redact before the write
Traces record the request, and the request is what the customer typed. Customers type email addresses, account numbers and sometimes card numbers into a support box without a second thought. Left alone, a trace store becomes the largest unreviewed collection of personal data in the company, copied into dashboards, exported to spreadsheets, retained for months.
The place to fix this is the writer, once, before any byte reaches disk. Apply the patterns to every attribute value and to the error string, replace matches with a marker such as [EMAIL] or [ACCOUNT], and recurse into lists. Redacting at the source means every downstream consumer is clean by construction. Redacting downstream means finding the copy you forgot about during the audit.
The marker tokens keep the traces useful. "[EMAIL] asked about a double charge" is still a complete story, and the timings, tokens, cost, scores and sources are untouched. Redaction removes the identity and keeps the evidence.
Turn the numbers into a gate
Someone proposes a friendlier system prompt: restate the question, list every step, add caveats, close with a summary. It reads well in review. It also makes every answer longer, and longer answers mean more completion tokens and more time per request. Nobody notices until the bill or the p95 alert.
A prompt is code, and it gets the same treatment as code: a fixed replay set, a baseline run, a candidate run, and a comparison of the two numbers a prompt or model change moves first, the p95 root latency and the mean cost per answer. Each gets an allowance. Past the allowance, the build fails with a table that says by how much:
metric baseline candidate change allowed result
p95_latency_ms 403.270000 403.199000 -0.0% +25% ok
mean_cost_usd 0.000102 0.000122 +20.0% +10% REGRESSION
GATE FAILED: candidate regresses beyond the allowance
Everything earlier in this guide feeds that gate. Spans give the timing, the usage block gives the cost, the percentile function gives the comparison, redaction lets the replay set live in the repository, and sampling means production keeps producing baselines for free.
What the tools add on top
Hosted and open-source products in this space, among them Langfuse, Arize Phoenix, MLflow Tracing, LangSmith and the LLM features of the general observability platforms, all consume the same shape of data: traces of spans with token usage and attributes. What they add is storage and search over millions of traces, a tree viewer, dashboards for the aggregates above, prompt versioning, and evaluation runs that score stored traces with a judge model or a rubric.
Choosing one is easier once you have built the layer yourself, because the questions become concrete. Does it accept OpenTelemetry spans, so your instrumentation is portable? Does sampling happen in your process, with your always-keep rules, or only after everything has been shipped to the vendor? Where does redaction run? Can a CI job read the aggregates back to gate a change? A team that has written a forty-line tracer answers those in an afternoon.
Build the layer yourself in about an hour
The LLM observability lab follows this article's order around a small support assistant with a real embedding model, a Milvus vector store and a chat model behind an in-cluster proxy. You stand the service up and read the code, write the span context manager that builds the tree, attach token usage and cost from the API's own numbers, produce the p50 and p95 report, find the wrong answer from retrieval scores alone, redact personal data in the writer, implement sampling with the always-keep rules and a budget alarm, and finish with the gate that catches a verbose prompt's cost regression. Every step has a checker that runs your code against the traces it produced, and the sandbox needs no local setup.
If you are new to the surrounding material, the MLOps course guide covers the loop the observability layer feeds, and the AI engineer roadmap places evaluation and monitoring in the wider skill set.