Evaluate an AI Agent: Trajectories, Tool Calls and an LLM Judge
Build the evaluation harness for a tool-using agent: record its trajectories on a golden set, grade final answers deterministically, score tool trajectories with exact, in-order and any-order matching plus precision and recall, check tool arguments, add an LLM judge with an order-swapped pairwise mode, measure the judge's agreement with human labels, run the suite into a report, and gate a prompt change on per-case regressions rather than the aggregate score.
Hands-on labs require Pro · $29.99/mo · cancel anytime
What you'll learn
- 1Record trajectories and read what an agent leaves behindThe system under test is agent.py, a finished tool-using agent for
- 2Grade the final answer without a modelThe four eval shapes lecture makes the ordering explicit: anything with a
- 3Score the tool trajectory against a reference planTwo runs can produce the same correct number by different routes. One
- 4Check the arguments, not just the tool namesCalling the right tool with the wrong arguments is the failure trajectory
- 5An LLM judge for the open part, with the order swappedNumbers, required phrases and tool paths are graded for free. What is left
- 6Calibrate the judge against human labelsThe pyramid in the eval-shapes lecture works in one direction: humans
- 7Run the suite into a reportFour graders and a judge, each answering one question about one case. The
- 8Gate the prompt change on per-case regressionsThe regression-tests lecture describes the trap exactly. A change fixes
Prerequisites
- Python: dicts, lists, json, regular expressions, a few lines of arithmetic
- The module's evaluation lectures: the four eval shapes, LLM-as-judge and its biases, regression tests for prompts
- Helpful: the tool-loop lab, whose finished agent is the system under test here
Exam domains covered
Skills & technologies you'll practice
This intermediate-level ai/ml lab gives you real-world reps across:
Why agents need their own evaluation
A single-turn model is graded on its answer. An agent is graded on its answer and on how it got there: which tools it called, in what order, with what arguments, in how many steps, and whether it stopped on a budget. A correct number reached without the lookup is a guess that will be wrong next week, and a right path that ends in a wrong number is a prompt problem. The harness in this lab scores both, with the cheap deterministic shapes where an answer is closed and a model judge where it is open. The recordings are real. The baseline trajectories come from a hosted Llama 3.3 70B model driving the agent over the golden set, failures included: on one case the model wrote its tool call out as text and never called anything. The candidate prompt is a shorter rewrite that reads fine in review. The final step compares the two per case, the way a golden-set regression test does, and the difference between the aggregate and the per-case view is the lesson.