TL;DR: LLM-as-a-judge means using a capable model to grade another model's output against a rubric, either on its own (a score) or against a second output (a preference). It is the only grader that scales for open-ended text, and it is the grader most likely to lie to you. The way to use it well is narrow: keep it off anything with a closed answer, give it a specific rubric and a reference, force JSON output and treat anything unparseable as no score, run every pairwise comparison twice with the order swapped, never let a model grade its own family, and check its verdicts against a small set of human labels before a single judge score reaches a dashboard. This guide covers each of those with the code shape, and the paired lab has you build and calibrate a judge inside an agent evaluation harness.
A team ships a support assistant and wants a quality number. Exact matching cannot score "was this explanation helpful", humans cannot read ten thousand transcripts a week, so someone asks a strong model to rate each answer from one to ten. The number goes on a dashboard. Three weeks later it has risen steadily, the team is pleased, and a customer forwards an answer that confidently describes a refund that never happened. The judge gave it a nine. It was long, polite and well structured.
That story contains every failure this guide is about, and every one of them has a known fix.
Build it, then read it
The agent evaluation lab has you write a JSON-only judge, an order-swapped pairwise judge, and the calibration against human labels, and run them live over recordings of a tool-using agent. It sits in the evaluation module of the AI Engineer course. This article is the reading that goes with it.
Where a judge belongs, and where it does not
Evaluation methods form a pyramid by cost and fidelity. Exact matching and parsing sit at the bottom: free, instant, objective, and limited to closed answers. Trained classifiers score one property cheaply at scale. A model judge is flexible and moderately expensive. Human review is the gold standard and does not scale. The mature setup uses all four, with cheap objective checks on every output and the judge only where nothing cheaper applies.
The consequence for a judge is a rule: it grades the open part of an answer and nothing else. Whether an answer contains 121.25, whether it says "unknown order", whether the agent called the lookup tool before quoting a price, are all questions a few lines of Python answer more reliably than any model. Sending them to a judge is slower, costlier and worse. What the judge is for is whether the explanation is faithful to what the tools returned, whether the answer claims an action that did not happen, whether one of two correct answers is clearer.
The research that made the pattern mainstream, Zheng and colleagues' "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (arXiv 2306.05685), found that a strong judge agreed with human preferences at roughly the rate humans agreed with each other. The same paper catalogued the biases below. Both findings hold: the judge is useful, and it is useful only with its failure modes handled.
Write the rubric like a spec
A vague instruction produces a noisy judge. "Rate this answer 1 to 10" gives different numbers on different days for the same answer, because the scale means nothing. A rubric that says what each level means, anchored to a reference answer, gives a judge something to check rather than something to feel:
Score the answer from 1 to 5 against the reference.
5: every fact and number matches the reference; nothing invented.
4: correct on the facts, with harmless extra or missing detail.
3: partly correct; one important fact missing or vague, none wrong.
2: a wrong number or a wrong claim next to correct ones.
1: wrong, or claims something happened that the reference says did not.
Judge the content only. Ignore length, tone and formatting.
Three properties matter. Every level names an observable condition. The reference answer gives the judge ground truth rather than asking it to know the answer itself. And the last line addresses a bias directly, which helps a little and costs nothing.
Force the output format
Ask for a number and a model will sometimes reply with a paragraph that contains two numbers. The fix is to ask for JSON only, parse it strictly, and record a missing score rather than a guessed one:
messages = [
{"role": "system", "content": "You are a strict, consistent grader. Reply with JSON only."},
{"role": "user", "content": f"{rubric}\n\nQuestion: {question}\n\nReference answer: {reference}\n\n"
f"Answer to grade: {answer}\n\n"
'Reply with exactly: {"score": <integer 1-5>, "reason": "<one sentence>"}'},
]
r = client.chat.completions.create(model=model, messages=messages, temperature=0, max_tokens=200)
data = extract_json(r.choices[0].message.content or "") or {}
score = data.get("score")
score = int(score) if isinstance(score, (int, float, str)) and str(score).isdigit() and 1 <= int(score) <= 5 else None
A None score is information: it tells you the judge did not follow the format on that input, and it keeps prose out of your averages. The one-sentence reason is not decoration either. It is what you read when the judge and a human disagree.
Pointwise or pairwise
A pointwise judge scores one answer on the rubric. A pairwise judge sees two answers to the same question and says which is better, or that they tie. Pairwise is the better instrument when the question is "did the new prompt improve things", because relative judgements are more stable than absolute scores, and it is the setting where the best-known bias lives.
The biases, and the mechanics that expose them
Position bias. In a pairwise comparison, judges tend to favour whichever answer is shown first, regardless of quality. The defence is mechanical: run every comparison twice with the order swapped, and keep a verdict only when both orders agree. A judge that prefers the first slot then produces inconsistent rather than a winner, which is the honest result:
first = ask(answer_a, answer_b) # judge says "1", "2" or "tie"
second = ask(answer_b, answer_a)
v1 = {"1": "A", "2": "B", "tie": "tie"}.get(first)
v2 = {"1": "B", "2": "A", "tie": "tie"}.get(second)
verdict = v1 if v1 is not None and v1 == v2 else "inconsistent"
Verbosity bias. Judges prefer longer answers, including ones whose extra length adds nothing. Instruct the judge to ignore length, compare answers of similar length where you can, and watch the calibration set for padded answers scoring above concise correct ones.
Self-enhancement bias. A judge rates outputs from its own model family higher. Grading a model with the same family as judge flatters it. Use a different family for the judge where possible, and treat a same-family judge's scores as relative rather than absolute.
Limited grading of reasoning. Judges are unreliable at checking arithmetic and multi-step reasoning inside an answer. Do not ask them to; parse the numbers and check them in code, and give the judge the reference so it compares rather than recomputes.
Argument blindness. When judges read agent trajectories rather than answers, research on trajectory judging finds they miss wrong tool arguments and undefined tool calls. Tool paths and arguments are checked deterministically, and the judge sees the final answer.
Calibrate before you trust a single score
The pyramid works in one direction: humans define what good means on a small sample, the judge is checked against them on that sample, and only then is the judge trusted at scale. The sample does not need to be large. It needs to contain the failure modes you care about: a correct concise answer, a correct padded one, a wrong number, a claimed action that was refused, an invented entity, a partial answer.
Score the sample with the judge and compare three ways. Accuracy of the pass/fail decision at a threshold (four and above passes, say) is the number you will use. Cohen's kappa on the same decision corrects for agreement that would happen by chance, which matters when most answers pass. The mean absolute difference of raw scores shows whether the judge is offset from the humans even when it ranks correctly.
hp = [h >= threshold for h in human_scores]
jp = [j >= threshold for j in judge_scores]
accuracy = sum(a == b for a, b in zip(hp, jp)) / n
p_h, p_j = sum(hp) / n, sum(jp) / n
expected = p_h * p_j + (1 - p_h) * (1 - p_j)
kappa = (accuracy - expected) / (1 - expected)
Then read the disagreements. They tell you whether the judge is lenient on invented success, harsh on padding, or reading the rubric differently from the person who wrote the labels. Fix the rubric, not the labels. In the lab's recorded run, the judge agreed with the human pass/fail decision on every one of ten labelled answers, with raw scores off by at most one point; that number is what earns it a place in the report.
Cost, sampling and where the judge sits in a pipeline
A judge call costs about as much as the call it grades. On every output of every run, that doubles the bill. Three habits keep it sane. Grade everything deterministic first and send only the open part to the judge. Sample in production, keeping every failure the cheap graders flagged and a fixed fraction of the rest, the same hash-based sampling used for traces. And re-run the calibration set whenever the judge model, the rubric or the reference answers change, since any of the three moves the numbers.
Judge failure modes and the fix for each
| Failure | What it looks like | Fix |
|---|---|---|
| Prose instead of a score | Averages drift; parse errors hidden | JSON-only reply, None on parse failure |
| Position bias | B wins when shown second | Ask both orders, keep only agreeing verdicts |
| Verbosity bias | Padded answers outscore concise correct ones | Rubric says ignore length; watch calibration set |
| Self-enhancement | Own family scores higher | Different-family judge; relative use only |
| Weak reasoning checks | Wrong arithmetic scored as correct | Parse numbers in code; give the judge a reference |
| Uncalibrated scores | A 9 for an invented refund | Human-labelled sample, accuracy and kappa, read the disagreements |
Build and calibrate a judge in the lab
The agent evaluation lab puts the judge where it belongs, as one grader among several in a harness for a tool-using agent. You write the JSON-only judge and the order-swapped pairwise judge, run them live against recorded answers, compute agreement with ten human labels, and then use the judge column in a report whose pass/fail decisions come from the deterministic graders. The wider picture, how trajectories and tool arguments are scored and how two prompt versions are compared per case, is in the AI agent evaluation guide.