Preporato
Evaluate a RAG Pipeline: Retrieval Metrics, Faithfulness and a Regression Gate
Hosted · ide
Beta

Evaluate a RAG Pipeline: Retrieval Metrics, Faithfulness and a Regression Gate

Turn 'the RAG bot seems better' into numbers you can gate on: chunk a knowledge base with paragraph provenance, label a golden set, score retrieval with recall, precision, MRR and nDCG, sweep chunk sizes, judge answer faithfulness claim by claim with a model, grade correctness, triage every failure by where it happened, and block a candidate configuration that regresses a protected question.

80 min8 steps3 domainsIntermediate
Part of the AI Engineer Course

Hands-on labs require Pro · $29.99/mo · cancel anytime

What you'll learn

  1. 1
    Chunk with provenance: every chunk knows which paragraphs it holds
    Northwind's support bot answers from a dozen help articles in kb/. The
  2. 2
    Recall@k and precision@k against paragraph labels
    runs/retrieval-s500.jsonl is a recorded retrieval run: for every
  3. 3
    Rank matters: MRR and nDCG
    Recall@3 says whether the right paragraph is somewhere in the top three.
  4. 4
    Sweep the chunk size and read the trade-off
    Four metrics per question is a spreadsheet; what the team needs is one
  5. 5
    Faithfulness: split the answer into claims and let a judge verify each one
    Retrieval metrics stop at the prompt. runs/baseline.jsonl continues:
  6. 6
    Correctness by rule, and a triage that names where each failure happened
    A faithful answer can still be wrong, and a wrong answer can be wrong for
  7. 7
    Run the suite: one report per configuration
    Everything so far scores one question. The team decides per
  8. 8
    The regression gate: block the change that breaks a protected question
    The team's proposal is in runs/candidate.jsonl: 1000-character chunks

Prerequisites

  • Python: functions, dicts, sets, json, list comprehensions
  • How a RAG pipeline works: chunk, embed, retrieve, generate (the module's earlier labs)
  • Read the module's two evaluation lectures first: recall and precision, then LLM-as-judge and failure modes

Exam domains covered

RAG SystemsLLM EvaluationLLM Operations

Skills & technologies you'll practice

This intermediate-level ai/ml lab gives you real-world reps across:

RAGRAG EvaluationLLM EvaluationLLM-as-a-JudgeRetrieval MetricsRegression Testing

Why evaluate RAG in two halves

A RAG system fails in two places. The retriever can miss the passage that holds the answer, and the generator can ignore or embellish the passages it was given. A single end-to-end score hides which one happened, and teams fix the wrong half. This lab measures each half with the metric that fits it and then combines them into a triage that names the failure. Retrieval is scored against paragraph-level labels with recall, precision, mean reciprocal rank and nDCG at k, and a chunk-size sweep shows the recall-versus-context trade-off with real numbers. Generation is scored by decomposing each answer into claims and asking a judge model whether the passages support each one, then by a deterministic check of the facts the answer must contain. A suite report rolls it up per question, and a regression gate compares a candidate configuration to the baseline and blocks it when a gated metric drops or a protected question stops passing.

Frequently asked questions

Why are the labels at the paragraph level instead of the chunk level?

Chunk ids change every time you change the chunk size, so chunk-level labels would have to be redone for every experiment. Paragraph ids are stable, chunks record which paragraphs they cover, and recall is scored on the paragraphs the retrieved chunks contain.

How is faithfulness measured?

The answer is split into sentence-level claims and a judge model is asked, with the retrieved passages in front of it, whether each claim is supported. Faithfulness is the supported fraction. A refusal asserts nothing and scores 1.0; correctness is graded separately, because a grounded answer can still be wrong or incomplete.

Which models does the lab use?

The lab's embedding model for retrieval, a hosted Llama 3.1 8B instruct model as the generator under test, and Llama 3.3 70B instruct as the judge, all through the lab's proxy. Retrieval runs, answers and judgments are recorded so the metric, triage, suite and gate steps run offline.

What does the regression gate check?

Mean recall and nDCG at k, mean faithfulness and the correct rate may not drop below the baseline, and every protected question that passed in the baseline must still pass in the candidate. The gate exits non-zero so a CI job can block the change.