Preporato
LLM Observability: Trace, Cost and Debug a RAG Assistant
Hosted · ide
Beta

LLM Observability: Trace, Cost and Debug a RAG Assistant

Instrument a real RAG assistant with traces from scratch: nested spans around retrieval and generation, token usage and cost attributed to every call, a latency and cost report, a triage tool that finds retrieval misses from trace evidence, PII redaction before anything is written, sampling that keeps every error and slow request, a cost budget alarm, and a CI gate that blocks a prompt change that makes the assistant slower or dearer.

75 min8 steps3 domainsIntermediate
Part of the AI Engineer Course

Hands-on labs require Pro · $29.99/mo · cancel anytime

What you'll learn

  1. 1
    Stand up the assistant and feel the blind spot
    You inherit the Northwind Cloud support assistant. It is a small but real
  2. 2
    Instrument the request with nested spans
    A trace is one request end to end. A span is one timed unit of work
  3. 3
    Attribute tokens and cost to every call
    The invoice arrives once a month as one number. The question that matters
  4. 4
    Report where the time and the money go
    Individual traces answer "what happened to this request". Operations questions
  5. 5
    Find the wrong answer from the trace, before the customer does
    Most wrong answers from a RAG assistant have the same anatomy. The question
  6. 6
    Redact personal data before it touches disk
    Traces record the request. The request is what the customer typed, and
  7. 7
    Keep every bad request, a tenth of the boring ones, and a budget alarm
    At six requests a run, keep everything. At a million requests a day,
  8. 8
    Gate a prompt change on latency and cost
    Someone proposes a friendlier system prompt: restate the question, list every

Prerequisites

  • Comfortable reading and editing Python
  • Have called a chat completions API at least once
  • Know what a percentile is

Exam domains covered

LLM ObservabilityEvaluation and MLOpsCost and Latency Engineering

Skills & technologies you'll practice

This intermediate-level ai/ml lab gives you real-world reps across:

LLM ObservabilityTracingLLMOpsCost MonitoringLatencyRAGEvaluationMilvus

What you'll do in this lab

This is a hands-on LLM observability lab on a real Retrieval-Augmented Generation (RAG) service: a Milvus vector store, NVIDIA embeddings and a chat model behind an OpenAI-compatible endpoint. You start with a working assistant that is completely opaque, then build the tracing layer yourself: a trace per request, nested spans for retrieval and generation with parent and child ids, status and error capture, and token usage read from the API's usage block and turned into cost per call and per answer from a price table you control. Every span is written as JSON you can query, and a small viewer prints each trace as a tree.

Then you use the traces the way an on-call engineer does. A report gives p50 and p95 latency per span and the share of time retrieval and generation each take, plus mean cost per answer. A triage tool finds the answers that were wrong because retrieval missed, from the cosine score the trace recorded rather than from reading transcripts. You redact emails and account numbers before anything touches disk, add sampling that keeps every error and every slow or expensive request while dropping most of the ordinary ones, wire a daily budget alarm, and finish with a CI gate that replays a fixed question set against a candidate prompt and fails the build when p95 latency or cost per answer regresses past an allowance.

Frequently asked questions

Do I need Langfuse, OpenTelemetry or MLflow for this lab?

No. You write the tracing layer with the Python standard library, which is the point: trace ids, span ids, parent links, attributes, status, sampling and redaction are the same concepts every tracing product exposes. Once you have implemented them once, adopting a vendor SDK is a thin wrapper over what you already understand.

Are the token counts and costs real?

Token counts come from the API's usage block on every live call, never from an estimate. Prices come from a small price table in the lab that uses illustrative figures; you replace them with your own provider's rate card and the same code gives you real cost per answer.

What is a retrieval miss and why does it show up in a trace?

A retrieval miss is a question the knowledge base does not cover, so the vector search returns the nearest chunks anyway and the model answers from noise. The retrieve span records the cosine score of the best chunk, so a low top score flags the miss from the trace alone, before anyone reads the answer.

Why sample traces at all?

At production volume, recording every span of every request costs more storage than it returns. The lab's sampler keeps every error, every slow request and every expensive request in full, and a fixed fraction of the ordinary ones, chosen by a hash of the trace id so the decision is reproducible.