TL;DR: An agent is graded on two things: its final answer and the path it took to get there. The answer is scored the cheap way wherever it has a closed form, with number parsing, required phrases and exact text. The path is scored against a reference plan: which tools were called, in what order, with what arguments, in how many steps. An LLM-as-a-judge covers only the open-text part, and it earns that role by replying in JSON, by being asked pairwise questions in both orders so position bias becomes visible, and by agreeing with a small set of human labels. The output of all of this is a report per prompt version, and the decision is made case by case: a change that raises the average while breaking a protected case does not ship. This guide walks through each piece with the code shape, and the paired lab has you build the whole harness around a real agent with real recordings.
Someone proposes a shorter system prompt for the operations agent. It reads better. In a quick test it answers the three questions everyone tries, and the answers look right. Two weeks after it ships, the agent starts doing unit conversions in its head instead of calling the converter, and one customer gets a weight that is off by a factor of two. Nothing in the quick test could have caught that, because the quick test looked at answers and the regression was in the path.
That is the difference between evaluating a model and evaluating an agent. A model produces text. An agent produces a trajectory: a sequence of tool calls with arguments and results, ending in text. Both parts can be right or wrong independently, and a harness that scores only one of them will pass changes that break the other.
Build it, then read it
The agent evaluation lab has you write every grader in this article and run them over real recordings of a tool-using agent, including a failure the baseline model produced on its own. It sits in the evaluation module of the AI Engineer course. This article is the reading that goes with it.
Start from the trajectory
Every grader reads from the same record: one trajectory per question, holding the final answer, the number of completions, every tool call with its raw arguments and its result, the tokens consumed, and whether a budget stopped the run. Reduce it once to the fields the graders need:
def summarize_trajectory(traj):
tools, args, errors = [], [], 0
for call in traj["tool_calls"]:
tools.append(call["name"])
try:
parsed = json.loads(call["arguments"])
args.append(parsed if isinstance(parsed, dict) else None)
except json.JSONDecodeError:
args.append(None)
if '"error"' in (call["result"] or ""):
errors += 1
return {"tools": tools, "args": args, "errors": errors, "steps": traj["steps"],
"tokens": traj["tokens"], "stopped_by": traj["stopped_by"],
"answer": traj["answer"] or ""}
Record trajectories on a fixed set of questions, the golden set, and keep the recordings. Two runs of the same model on the same prompt will differ, sometimes in the path, occasionally in whether the model calls a tool at all. In the lab's baseline recording the model wrote a tool call out as text on one question and never called anything. The harness has to catch that, and the recording is what makes the catch reproducible.
The golden set: what a correct run looks like
Each case states three kinds of expectation, and a case is only as useful as the expectations are specific.
What a golden-set case specifies
| Part | What it states | Example |
|---|---|---|
| Answer | Numbers within a tolerance, required phrases, or exact text | 121.25 within 0.01; one of unknown, not found, cannot |
| Trajectory | The tools a correct run calls, and how strictly to match | lookup_order then shipping_quote, in order |
| Arguments | Parameters of the first call of a tool | shipping_quote with country JP and weight_kg about 11.5 |
| Budget | A ceiling on completions; a budget stop always fails | at most 5 steps |
| Reference | A model answer for the judge | Shipping order NW-1007 to Japan costs 121.25 USD |
Mark the cases nobody may break as protected: past bugs, the core flows, the case a customer complained about. That flag is what the gate at the end enforces.
Grade the answer without a model
The evaluation shapes form a pyramid. Exact matching and parsing sit at the bottom: free, instant, objective, and limited to closed answers. Most of what an operations agent says is a closed answer, so the first grader is deterministic and has to cope with how a correct number gets written: $121.25 USD, 1,280.00, 27.78 pounds against a reference of 27.778.
NUM_RE = re.compile(r"-?\d{1,3}(?:,\d{3})+(?:\.\d+)?|-?\d+(?:\.\d+)?")
def grade_answer(answer, spec):
reasons = []
text = normalize(answer)
for group in spec.get("contains_any", []):
if not any(normalize(alt) in text for alt in group):
reasons.append(f"answer lacks any of {group}")
if "numbers" in spec:
found = [float(x.replace(",", "")) for x in NUM_RE.findall(answer)]
for want in spec["numbers"]:
if not any(abs(got - want) <= spec["tolerance"] for got in found):
reasons.append(f"number {want} not in answer (found {found})")
return not reasons, reasons
Spending a judge on whether the answer says 121.25 is slower, dearer and less reliable than this. Save the judge for what this cannot see.
Score the trajectory against a reference plan
Two runs can reach the same number by different routes, and only one of them generalises. The reference plan lists the tools a correct run calls; a matching mode says how strictly to compare.
- Exact means the same list, for paths that are fully determined.
- In order means the reference appears as a subsequence of the actual calls. Extra calls are tolerated, missing or reordered ones are not, with partial credit for the share matched in order.
- Any order means every reference tool was called, in any sequence.
Alongside the score, tool-selection precision (calls that were expected, over all calls) and recall (expected tools present, over expected tools) and a count of extra calls describe the path even when the score is perfect. A reference can list alternative plans, since converting a weight through the calculator is as good as converting it through the unit tool, and the trajectory takes the best score across them.
def in_order_score(actual, expected):
i = 0
for tool in actual:
if i < len(expected) and tool == expected[i]:
i += 1
return i / len(expected)
In the lab's candidate recording, the shortened prompt produced the right difference between two item weights without calling the calculator. The answer grader passed it. The trajectory score was 0.5 with recall 0.5, and that row is the regression the quick test would have missed.
Check the arguments, not just the tool names
Calling the right tool with the wrong arguments is invisible to trajectory matching. A quote for the right order to the wrong country, a refund of the right amount on the wrong order, a list of orders without the status filter followed by the model counting by eye. Research on judges that read trajectories has a name for their weakness here, argument blindness, so the check is deterministic: for the first parsable call of each named tool, compare each parameter as a plain value, as a number within a tolerance, or against a set of options, and name the tool, the parameter and the value seen in every reason.
The judge, made honest
What remains is the open part: did the answer say the refund was refused, or did it imply money moved; is the explanation faithful to what the tools returned. That is the LLM-as-a-judge shape, and it comes with known biases. Position bias makes a judge prefer whichever answer it saw first. Length bias rewards padding. A judge from the same model family flatters its relatives. And a judge asked for a score often replies with prose.
Two mechanics handle most of it. The single-answer judge gets a narrow rubric, the reference answer, and an instruction to reply with JSON only; anything that does not parse scores None, never a guessed number:
messages = [
{"role": "system", "content": "You are a strict, consistent grader. Reply with JSON only."},
{"role": "user", "content": f"{rubric}\n\nQuestion: {question}\n\nReference answer: {reference}\n\n"
f"Answer to grade: {answer}\n\n"
'Reply with exactly: {"score": <integer 1-5>, "reason": "<one sentence>"}'},
]
The pairwise judge asks which of two answers is better, twice, with the answers swapped between the two calls, and keeps a verdict only when both orders agree. A judge that prefers whichever answer came first shows up as inconsistent, which is the honest result and the one the lecture on judge biases prescribes.
Calibrate before you trust
The pyramid works in one direction. Humans define what good means on a small sample; the judge is checked against them on that sample; only then is the judge trusted at scale. Ten labelled answers are enough to start: correct and concise, correct and padded, a wrong number, a claimed refund that was refused, an invented order. Score the same ten with the judge and measure agreement three ways: accuracy of the pass/fail decision at a threshold, Cohen's kappa on that decision to correct for chance, and the mean absolute difference of the raw scores. Then read the disagreements, because they show whether the judge is lenient on invented success or harsh on padding, and fix the rubric rather than the labels.
The report, and the decision
The suite runner asks every grader about every case and writes one report: a row per case with the answer grade, trajectory score, precision and recall, argument check, steps, tokens, tool errors, the judge's score, and the reasons when anything failed. A case passes when the answer and the arguments are right, the trajectory score is 1.0, no budget stopped the run, and it stayed within its step ceiling. The top of the report carries the aggregates a dashboard wants.
The top of the report is also the part not to decide from. A change fixes some cases and breaks others, and the aggregate nets them. The gate compares two reports case by case:
baseline pass_rate=0.9 candidate pass_rate=0.8
mean_trajectory_score +0.05
mean_precision +0.1334
mean_steps -0.2
total_tokens -4217
regressions (pass -> fail): ['heavier-item', 'price-and-ship-in']
fixes (fail -> pass): ['convert-two']
protected regressions: ['heavier-item']
GATE FAILED
Every aggregate except the pass rate favours the candidate. It fixed the case the baseline failed. It also broke two, one of them protected, and a protected case going from pass to fail blocks the change whatever the totals say. The review is now about three cases rather than two numbers, which is a review a person can actually do.
Build the harness yourself in about eighty minutes
The agent evaluation lab follows this article's order over a real tool-using agent and its recordings. You record trajectories on the golden set, write the answer grader, the trajectory scorer with its three matching modes, the argument checker, the JSON-only judge and its order-swapped pairwise mode, the calibration against ten human labels, the suite runner, and the per-case gate, and you finish by watching a shorter prompt get blocked on a protected regression while every other aggregate says it is an improvement. Each step has a checker that runs your code, and the sandbox needs no local setup.
The agent under test is the one built in the tool-loop lab, and the traces that make the report possible follow the same shape as the spans in the LLM observability guide.