Preporato

hands-on labIDE in your browser

Evaluate an AI Agent: Trajectories, Tool Calls and an LLM Judge

Build the evaluation harness for a tool-using agent: record its trajectories on a golden set, grade final answers deterministically, score tool trajectories with exact, in-order and any-order matching plus precision and recall, check tool arguments, add an LLM judge with an order-swapped pairwise mode, measure the judge's agreement with human labels, run the suite into a report, and gate a prompt change on per-case regressions rather than the aggregate score.

time
80 min
checked steps
8
level
Intermediate
setup
None
NCP-AAINCA-AMADPart of the AI Engineer Course

$29.99a month with Pro

Pro opens all 8 steps of this lab and every other lab. Cancel any time.

no account needed · Pro opens all 8 steps

See Pro plans
Lab cockpit80 min · 8 stepsSession running
7 / 8 steps passingGate the prompt change on per-case regressions · step 8 of 8
harness.py▶ Run✓ Check
# ----------------------------------------------------------------- Step 8def compare_reports(base, cand, max_pass_rate_drop=0.0):    """Per-case comparison of two suite reports. Returns {"regressions": [ids that    went pass -> fail], "fixes": [fail -> pass], "protected_regressions": [...],    "deltas": {metric: cand - base}, "ok"}. ok requires no regression on a protected    case and a pass-rate drop no larger than max_pass_rate_drop."""
TerminalOutput
step 8 of 8, replayed: Gate the prompt change on per-case regressions. The code, the output and the check are the lab's own.

The job

Northwind Cloud's operations agent answers questions like "what would it cost to ship order NW-1041 to its destination" by calling order lookups, a calculator, a unit converter, a shipping-rate tool and a guarded refund tool. Someone proposes a shorter system prompt. Nobody can say whether it is safe, because nobody can say what "correct" means for an agent beyond "the answer looked fine". You build the harness that can: a golden set of ten cases with expected answers, expected tool paths and expected arguments; deterministic graders for each; a model judge for the open-text part, checked for position bias and calibrated against human labels; a suite runner; and a gate that compares two prompts case by case, so a regression on a protected case blocks the change even when the aggregate score goes up.

8 steps, each checked when you finish it

A check runs your work at the end of every step. Hints and the full solution are there if you get stuck.

  1. 1

    Record trajectories and read what an agent leaves behind

    The system under test is agent.py, a finished tool-using agent for Northwind's operations desk: order lookup, a calculator, a unit converter, shipping quotes and a guarded refund tool, driven by a hosted 70B model.

    read this step below ↓

  2. 2

    Grade the final answer without a model

    The four eval shapes lecture makes the ordering explicit: anything with a definite right answer is graded with exact matching or a parser, because that is free, instant and cannot be biased.

  3. 3

    Score the tool trajectory against a reference plan

    Two runs can produce the same correct number by different routes.

  4. 4

    Check the arguments, not just the tool names

    Calling the right tool with the wrong arguments is the failure trajectory matching cannot see.

  5. 5

    An LLM judge for the open part, with the order swapped

    Numbers, required phrases and tool paths are graded for free.

  6. 6

    Calibrate the judge against human labels

    The pyramid in the eval-shapes lecture works in one direction: humans define what good means on a small sample, the judge is checked against them on that sample, and only then is the judge trusted at scale.

  7. 7

    Run the suite into a report

    Four graders and a judge, each answering one question about one case.

  8. 8

    Gate the prompt change on per-case regressions

    The regression-tests lecture describes the trap exactly.

Step 1 as it appears in the lab

The lab’s own text. The hint and the solution stay inside the lab.

Step 1: Record trajectories and read what an agent leaves behind

The system under test is agent.py, a finished tool-using agent for Northwind's operations desk: order lookup, a calculator, a unit converter, shipping quotes and a guarded refund tool, driven by a hosted 70B model. cases.jsonl is its golden set: ten questions, each with what a correct run looks like. record.py runs the agent over the cases and writes one trajectory per case: the final answer, the number of completions, every tool call with its arguments and result, token usage, and whether a budget stopped the run.

A single-turn model is graded on its answer. An agent is graded on its answer and on the path: a right number reached without the lookup is a guess that will be wrong next week, and a right path ending in a wrong number is a prompt problem. Everything in this lab reads from the trajectory, so the first job is reducing a recorded run to the fields the graders need.

do this

1. Run. The live recording covers three cases so it stays quick. Then harness.py summarize prints the shipped baseline recording, trajectories-baseline.jsonl, which was made the same way over all ten cases. Read it closely. On convert-two the model wrote its tool call out as text and never called anything, so the answer is a JSON blob. That is a real failure in the baseline, kept on purpose.

2. Implement summarize_trajectory(traj) in harness.py. From one trajectory dict return {"id", "tools", "args", "errors", "steps", "tokens", "stopped_by", "answer"}: tools is the tool names in call order, args holds json.loads of each call's arguments (or None when the arguments are not a JSON object), errors counts tool results that contain "error", and answer is the answer text or "".

3. Run again. The summary shows each case's tool path, error count and answer. Compare trajectories-live.jsonl with the baseline for the same three cases: same model, same prompt, and the paths may already differ. That variance is why the rest of the lab scores paths with tolerance and gates on per-case flips rather than single runs.

Why this matters in practice

Every evaluation framework for agents starts from this reduction. The trajectory is the ground truth; the graders are functions of it; the report is a table of those functions over a fixed set.

provided for you:agent.pycases.jsonlfake_client.pyorders.jsonrecord.pytools.pytrajectories-baseline.jsonl

what you get

  • An editor and terminal in your browser
  • A check at the end of every step
  • Hints and the full solution
  • Nothing to install or configure

before you start

  • Python: dicts, lists, json, regular expressions, a few lines of arithmetic
  • The module's evaluation lectures: the four eval shapes, LLM-as-judge and its biases, regression tests for prompts
  • Helpful: the tool-loop lab, whose finished agent is the system under test here

Exam domains covered

Agent EvaluationEvaluation and MLOpsTool Calling and Agents

Part of the AI agents and MCP track

Lab 20 of 26. Tool loops, ReAct, MCP servers and clients, multi-agent supervisors and agent evaluation.

See the whole track
Intermediate · 80 minBuild an Agent From Scratch: The Tool Loop Without a FrameworkIn browserfree
Intermediate · 35 minBuild a ReAct Agent with NVIDIA NIMIn browserPro
Intermediate · 35 minBuild an AI Agent 3 Ways: ReAct vs Tool Calling vs Plan-and-ExecuteIn browserPro
Intermediate · 30 minStructured Output & Function Calling with NIMIn browserPro
Intermediate · 50 minWeb Research Agent: Search, Read and Cite Without Being Fooled by the WebIn browserPro
Intermediate · 50 minCode-Executing Data Agent: Let an LLM Answer Questions With pandas, SafelyIn browserPro
Intermediate · 50 minText-to-SQL Agent: Answer Questions From a Database Without Letting the Model Touch the DataIn browserPro
Intermediate · 55 minSpec to Code: Build a Test-Driven Coding AgentIn browserPro
Intermediate · 50 minAI Code Review Bot: Parse a Diff, Find Bugs, Score and Gate the PRIn browserPro
Intermediate · 55 minRefactor Legacy Code Safely: Characterization Tests and an AI Refactoring AgentIn browserPro
Intermediate · 55 minDebug With an Agent: Reproduce, Fix, and Add the Regression TestIn browserPro
Intermediate · 35 minAdd Long-Term Memory to an AI Agent: LangGraph + MilvusIn browserPro
Advanced · 40 minBuild an MCP Tool Server & Connect a LangChain AgentIn browserPro
Intermediate · 60 minBuild an MCP Client and Host: Handshake, Tool Discovery, Error Handling and ApprovalsIn browserPro
Advanced · 40 minBuild Two Agents That Talk via the A2A ProtocolIn browserPro
Intermediate · 40 minBuild a Multi-Agent Supervisor with LangGraphIn browserPro
Intermediate · 60 minPlanner-Executor Agents with Self-Critique: Plans, Validators, Replanning and Stopping RulesIn browserPro
Intermediate · 60 minHuman-in-the-Loop Agent: Approval Interrupts, Durable Pauses, Crash-Safe Tools and EscalationIn browserPro
Intermediate · 30 minEvaluate an Agent with LLM-as-JudgeIn browserPro
this labEvaluate an AI Agent: Trajectories, Tool Calls and an LLM JudgeIn browserPro
Intermediate · 25 minModel Routing & Cost Cascade with NIMIn browserPro
Intermediate · 60 minOptimise Prompts with DSPy: Signatures, Metrics, Few-Shot Compilers and Held-Out TestsIn browserPro
Beginner · 45 minGate an Agent's Tool Calls with JevIn browserfree
Intermediate · 60 minRoute Between a Small and a Large Model with JevIn browserPro
Intermediate · 60 minCalibrate Jev's Probabilities and ThresholdsIn browserPro
Intermediate · 50 minFind Where Jev Fails Before Your Users DoIn browserPro

read first

Questions about this lab

What is the difference between evaluating an LLM and evaluating an agent?

An agent produces a trajectory as well as an answer: the tools it called, the arguments, the order, the step count. Agent evaluation scores the trajectory against a reference plan alongside the final answer, because a right answer reached the wrong way does not generalise.

How do you score a tool trajectory?

Against a reference plan with a matching mode: exact (same tool list), in-order (the reference appears as a subsequence, with partial credit), or any-order (the reference multiset is covered). Tool-selection precision and recall and a count of extra calls complete the picture; the lab implements all of them.

When should an LLM judge be used instead of exact matching?

Only for the open-text part of an answer. Numbers, required phrases and tool paths are graded deterministically, which is cheaper and cannot be biased. The judge grades against a rubric, pairwise comparisons run in both orders to expose position bias, and its scores are checked against human labels before they are trusted.

Why compare prompts per case instead of by average score?

Because an aggregate nets improvements against regressions. A candidate prompt can raise the pass rate while breaking cases users rely on. The gate in this lab lists every case that flipped and fails on any regression in a protected case.

Why agents need their own evaluation

A single-turn model is graded on its answer. An agent is graded on its answer and on how it got there: which tools it called, in what order, with what arguments, in how many steps, and whether it stopped on a budget. A correct number reached without the lookup is a guess that will be wrong next week, and a right path that ends in a wrong number is a prompt problem. The harness in this lab scores both, with the cheap deterministic shapes where an answer is closed and a model judge where it is open. The recordings are real. The baseline trajectories come from a hosted Llama 3.3 70B model driving the agent over the golden set, failures included: on one case the model wrote its tool call out as text and never called anything. The candidate prompt is a shorter rewrite that reads fine in review. The final step compares the two per case, the way a golden-set regression test does, and the difference between the aggregate and the per-case view is the lesson.