Preporato

hands-on labIDE in your browser

LLM Observability: Trace, Cost and Debug a RAG Assistant

Instrument a real RAG assistant with traces from scratch: nested spans around retrieval and generation, token usage and cost attributed to every call, a latency and cost report, a triage tool that finds retrieval misses from trace evidence, PII redaction before anything is written, sampling that keeps every error and slow request, a cost budget alarm, and a CI gate that blocks a prompt change that makes the assistant slower or dearer.

time
75 min
checked steps
8
level
Intermediate
setup
None
NCP-AAINCA-AMADPart of the AI Engineer Course

$29.99a month with Pro

Pro opens all 8 steps of this lab and every other lab. Cancel any time.

no account needed · Pro opens all 8 steps

See Pro plans
Lab cockpit75 min · 8 stepsSession running
6 / 8 steps passingKeep every bad request, a tenth of the boring ones, and a budget alarm · step 7 of 8
budget.py▶ Run✓ Check
import sys from cost import load_pricingfrom tracer import load_traces  def spend(traces):    """Return (total_usd, [(cost, input) ...] sorted most expensive first)."""
TerminalOutput
step 7 of 8, replayed: Keep every bad request, a tenth of the boring ones, and a budget alarm. The code, the output and the check are the lab's own.

The job

You own the Northwind Cloud support assistant, a small RAG service: a knowledge base in a Milvus vector store, NVIDIA embeddings, and a chat model behind the in-cluster proxy. Answers are usually fine. Nobody can say how long a request takes, where the time goes, what a day of traffic costs, or which answers were wrong before a customer says so. You fix that in eight steps, each one a piece of the observability layer real teams run: nested spans, token and cost attribution, a latency report, trace-driven triage of a bad answer, redaction, sampling, a budget alarm, and a regression gate. You write the tracing layer yourself with the standard library, so when you pick up OpenTelemetry, Langfuse or MLflow Tracing afterwards, every concept is already yours.

8 steps, each checked when you finish it

A check runs your work at the end of every step. Hints and the full solution are there if you get stuck.

  1. 1

    Stand up the assistant and feel the blind spot

    You inherit the Northwind Cloud support assistant.

    read this step below ↓

  2. 2

    Instrument the request with nested spans

    A trace is one request end to end.

  3. 3

    Attribute tokens and cost to every call

    The invoice arrives once a month as one number.

  4. 4

    Report where the time and the money go

    Individual traces answer "what happened to this request".

  5. 5

    Find the wrong answer from the trace, before the customer does

    Most wrong answers from a RAG assistant have the same anatomy.

  6. 6

    Redact personal data before it touches disk

    Traces record the request.

  7. 7

    Keep every bad request, a tenth of the boring ones, and a budget alarm

    At six requests a run, keep everything.

  8. 8

    Gate a prompt change on latency and cost

    Someone proposes a friendlier system prompt: restate the question, list every step, add caveats, close with a summary.

Step 1 as it appears in the lab

The lab’s own text. The hint and the solution stay inside the lab.

Step 1: Stand up the assistant and feel the blind spot

You inherit the Northwind Cloud support assistant. It is a small but real Retrieval-Augmented Generation (RAG) service: eight knowledge-base documents, chunked and embedded with an NVIDIA embedding model through the in-cluster proxy, stored in a Milvus vector store, retrieved by cosine similarity, and turned into an answer by a chat model behind the same proxy.

It works. Ask it how to reset a password and it tells you. What it cannot tell you is anything about itself: how long a request took, whether the time went to retrieval or to the model, how many tokens the model consumed, what a day of traffic costs, or which of today's answers were wrong. Every one of those questions arrives eventually, usually from finance or from an angry customer, and today the only honest answer is "we don't know".

That gap is what observability closes, and this lab has you close it by hand. Before instrumenting anything, run the service and read the code you are about to instrument.

do this

1. Build the index and answer the question set. Click Run. The first command embeds the knowledge base and writes the Milvus collection to /home/labuser/kb.db. The second answers the six questions in questions.jsonl, prints each answer with the documents it retrieved, and writes answers.jsonl.

2. Read app.py top to bottom. Three functions matter for the rest of the lab:

  • retrieve(question) embeds the question and runs a top-k vector search. It returns chunks with their cosine scores.
  • generate(question, hits) builds the prompt from the retrieved chunks and calls the chat model. The API response carries a usage block with the real prompt and completion token counts.
  • answer(question) is the request end to end: retrieve, then generate.

Notice the with TRACER.span(...) lines already wrapped around each of them. Those are the instrumentation points. They call into tracer.py, which right now is a stub that records nothing. app.py decides where the spans are; tracer.py decides what is recorded, redacted, kept and priced. Keeping those two decisions in separate files is the first design choice of an observability layer, because the call sites rarely change and the policy changes every quarter.

3. Look at answers.jsonl. It has the answer and the sources for each question and nothing else. No timing, no tokens, no cost, no trace id. That is the blind spot you are about to remove.

Why this matters in practice

Teams typically add observability after the first incident, when it is too late to know what normal looked like. The order in this lab is the order that works: instrument first, so the baseline exists before anyone needs it.

provided for you:cost.pykb_account-deletion.mdkb_api-keys.mdkb_billing.mdkb_data-export.mdkb_getting-started.mdkb_regions.mdkb_sso.mdkb_status-incidents.mdpricing.yamlquestions.jsonltracer.py

what you get

  • An editor and terminal in your browser
  • A check at the end of every step
  • Hints and the full solution
  • Nothing to install or configure

before you start

  • Comfortable reading and editing Python
  • Have called a chat completions API at least once
  • Know what a percentile is

Exam domains covered

LLM ObservabilityEvaluation and MLOpsCost and Latency Engineering

Part of the LLMOps and MLOps track

Lab 20 of 22. vLLM serving, load tests against SLOs, tracing, drift monitoring and prompt tests in CI.

See the whole track
Intermediate · 45 minDeploy & Serve LLMs in Production (Jupyter)NVIDIA GPUPro
Intermediate · 40 minInference Serving Patterns: Dynamic Batching, Throughput, and the Triton Mental ModelNVIDIA GPUPro
Advanced · 55 minvLLM Production Serving: PagedAttention, Continuous Batching, Prefix CachingNVIDIA GPUPro
Intermediate · 40 minBatch Size & Precision Sweep: Finding Your Sweet SpotNVIDIA GPUPro
Intermediate · 60 minServe an ML Model on CPU: ONNX Export, FastAPI, Dynamic Batching and a p95 Latency BudgetIn browserPro
Intermediate · 55 minShip an LLM Backend: FastAPI with API Keys, Rate Limits, Quotas, Streaming and Safe FailuresIn browserPro
Intermediate · 90 minBuild an LLM Gateway: One Door Between Your Apps and the ModelIn browserPro
Intermediate · 50 minStreaming and Prompt Caching: Time to First Token, SSE and Cheaper PromptsIn browserPro
Intermediate · 50 minContext Engineering: Token Budgets, Compaction and Running Notes for Long ChatsIn browserPro
Intermediate · 45 minEvaluation & Benchmarking LLMsNVIDIA GPUPro
Intermediate · 50 minPrompt Regression Testing in CI: A Golden Set That Blocks Bad Prompt ChangesIn browserPro
Intermediate · 50 minCI for Models: Data Contracts, Behavioral Tests, Golden Sets and a Merge GateIn browserPro
Intermediate · 45 minTraining Pipeline Orchestration: Ordering, Retries, Failure Isolation and ResumeIn browserPro
Intermediate · 45 minData and Model Versioning: Content Addressing, Lineage and Garbage CollectionIn browserPro
Intermediate · 60 minFeature Store Basics: Point-in-Time Correctness and Training-Serving SkewIn browserPro
Intermediate · 45 minPublish a Model: Card, Evaluation and a Versioned ReleaseIn browserPro
Intermediate · 35 minMLflow Experiment Tracking: From Single Run to Team WorkflowNVIDIA GPUPro
Intermediate · 40 minReproducible Training: The Flags, The Cost, The ArtifactsNVIDIA GPUPro
Intermediate · 50 minLLM Load Testing and SLOs: TTFT, TPOT, Open-Loop Traffic, Goodput and CapacityIn browserPro
this labLLM Observability: Trace, Cost and Debug a RAG AssistantIn browserPro
Intermediate · 50 minML Model Monitoring: Data Drift with PSI, Delayed Labels, Alert Rules and Root CauseIn browserPro
Intermediate · 45 minIncident Drill: Debug a Failing ML Service from Logs and MetricsIn browserPro

read first

Questions about this lab

Do I need Langfuse, OpenTelemetry or MLflow for this lab?

No. You write the tracing layer with the Python standard library, which is the point: trace ids, span ids, parent links, attributes, status, sampling and redaction are the same concepts every tracing product exposes. Once you have implemented them once, adopting a vendor SDK is a thin wrapper over what you already understand.

Are the token counts and costs real?

Token counts come from the API's usage block on every live call, never from an estimate. Prices come from a small price table in the lab that uses illustrative figures; you replace them with your own provider's rate card and the same code gives you real cost per answer.

What is a retrieval miss and why does it show up in a trace?

A retrieval miss is a question the knowledge base does not cover, so the vector search returns the nearest chunks anyway and the model answers from noise. The retrieve span records the cosine score of the best chunk, so a low top score flags the miss from the trace alone, before anyone reads the answer.

Why sample traces at all?

At production volume, recording every span of every request costs more storage than it returns. The lab's sampler keeps every error, every slow request and every expensive request in full, and a fixed fraction of the ordinary ones, chosen by a hash of the trace id so the decision is reproducible.

What you'll do in this lab

This is a hands-on LLM observability lab on a real Retrieval-Augmented Generation (RAG) service: a Milvus vector store, NVIDIA embeddings and a chat model behind an OpenAI-compatible endpoint. You start with a working assistant that is completely opaque, then build the tracing layer yourself: a trace per request, nested spans for retrieval and generation with parent and child ids, status and error capture, and token usage read from the API's usage block and turned into cost per call and per answer from a price table you control. Every span is written as JSON you can query, and a small viewer prints each trace as a tree.

Then you use the traces the way an on-call engineer does. A report gives p50 and p95 latency per span and the share of time retrieval and generation each take, plus mean cost per answer. A triage tool finds the answers that were wrong because retrieval missed, from the cosine score the trace recorded rather than from reading transcripts. You redact emails and account numbers before anything touches disk, add sampling that keeps every error and every slow or expensive request while dropping most of the ordinary ones, wire a daily budget alarm, and finish with a CI gate that replays a fixed question set against a candidate prompt and fails the build when p95 latency or cost per answer regresses past an allowance.