TL;DR: There is no single LLM evaluation metric, because an LLM system fails in several places and each place needs its own number. Retrieval is graded with recall, precision, mean reciprocal rank and nDCG against labelled passages. Answers are graded against a reference with exact or rule-based matches, and against the passages they were given with faithfulness, the fraction of claims the context supports. Agents are graded on their tool trajectories and arguments as well as their final answer. Judges are graded on agreement with human labels. Operations are graded on p95 latency and cost per request. Pick the metric by the failure you are trying to catch, keep the ones that disagree with each other, and gate on the rows, never only on the means.
A team ships a support bot that answers from a knowledge base. The dashboard shows one number, an overall quality score from a judge model, and it went up last week. Users say the bot got worse. Both are true: the new chunking made the answers more fluent and better grounded in whatever was retrieved, and it also stopped retrieving the paragraph that answers the most common question. One number averaged a gain and a loss into a gain. The fix was four numbers.
Build the harness
The RAG evaluation lab in the AI Engineer course implements most of the metrics below over a real pipeline: paragraph-level labels, retrieval metrics that survive re-chunking, a claim-level faithfulness judge, a rule-based correctness grade, a triage that names where each failure happened, and a regression gate. The agent evaluation lab covers the trajectory and judge-calibration metrics.
Metrics are organised by what they compare
Every evaluation metric compares two things. Once you name the two things, the metric follows, and so does the failure it can and cannot see.
LLM evaluation metrics by what they compare
| What is compared | Metrics | Failure it catches | How it lies |
|---|---|---|---|
| Retrieved passages vs labelled passages | recall@k, precision@k, MRR, nDCG@k | The right paragraph never reached the prompt | Labels on chunk ids break when chunking changes; label paragraphs instead |
| Answer vs reference answer | exact match, rule-based fact checks, F1 or ROUGE on tokens, semantic similarity | Wrong or incomplete answer | Token overlap rewards paraphrase of the wrong thing; rules need alternatives |
| Answer vs retrieved context | faithfulness (supported claims / all claims), citation accuracy | Hallucination, embellishment | A grounded answer can still be wrong if the context is wrong or stale |
| Answer vs question | answer relevance, refusal correctness | Off-topic answers; answering the unanswerable; refusing the answerable | Relevance is a judge call and inherits judge bias |
| Agent trajectory vs expected plan | tool precision and recall, argument accuracy, step count, budget stops | Right answer by the wrong route; wasted or dangerous calls | Alternative valid plans need to be labelled, or the metric punishes them |
| Judge verdicts vs human labels | accuracy, Cohen's kappa, MAE, position-swap consistency | A judge that rubber-stamps or that prefers length and its own style | Kappa on ten labels is noise; calibrate on dozens |
| Request vs its budget | p95 latency per stage, tokens, cost per request, cost per resolved case | Regressions that pass every quality gate and double the bill | Means hide tails; sample so failures are always kept |
The table is also the order to build in. Retrieval metrics are cheap and deterministic. Reference-based checks are deterministic once the labels exist. Faithfulness and relevance need a judge, which costs money and needs calibrating. Trajectory metrics need recorded runs. Operational metrics need tracing. Each layer catches something the previous one cannot, and none replaces the ones before it.
Retrieval metrics
Recall@k is the fraction of relevant passages that appear in the top k retrieved chunks. It is the ceiling on the whole system: a passage that never reaches the prompt cannot be used. Precision@k is the fraction of the top k chunks that are relevant, which is the price you pay in tokens and attention for that recall. The two move against each other as k changes, and the standard production pattern is to retrieve at a generous k for recall and rerank down to a few passages for precision.
Mean reciprocal rank looks only at where the first relevant chunk sits: 1 for rank one, 0.5 for rank two, and so on. It is the metric to watch before lowering k. nDCG@k scores the whole ranking with a log discount by position, normalised by the ideal ranking, and it is the metric that moves when a reranker, an embedding model or hybrid retrieval changes the order without changing what is found.
Two details decide whether these numbers mean anything. Label relevance at a level that survives your experiments: paragraph or section ids rather than chunk ids, with each chunk recording which paragraphs it covers. And decide what counts as a hit when one paragraph is split into fragments, or nDCG will exceed 1.0 and recall will be flattered by duplicates.
Reference-based answer metrics
When a question has a known answer, grade against it. Exact match works for short factual answers and nothing else. Rule-based fact checks are the workhorse for support and operations content: a list of facts the answer must contain, with alternatives, and a list of phrases it must not contain because they mark a confusion with a neighbouring passage. They are deterministic, cheap, and readable by the people who write the knowledge base.
Token-overlap scores such as F1 and ROUGE compare word sets with the reference and are inherited from summarisation research. They reward paraphrase of the wrong thing and punish a correct answer phrased differently, so treat them as a smoke test. Semantic similarity with an embedding model fixes the phrasing problem and introduces a threshold you have to tune, and it cannot tell "30 minutes" from "30 days".
For questions the system should refuse, the reference is the refusal itself. A correct refusal on an unanswerable question and a wrong refusal on an answerable one are both worth counting, because a model that learns to refuse more will improve faithfulness while getting less useful.
Context-based answer metrics
Faithfulness, also called groundedness, is the fraction of an answer's claims that the retrieved context supports. The recipe that works is decompose then verify: split the answer into sentence-level claims, show a judge model the passages and the numbered claims, and get one verdict per claim. The fraction is the score; the flagged claim is the debugging aid. Three practical rules from running it: strip citation markers from claims before judging, because a claim that names "passage [2]" sends the judge to that passage alone; a refusal asserts nothing and scores 1.0; and an unparseable judge reply must count as unsupported rather than pass silently.
Faithfulness and correctness are different questions. An answer can be entirely supported by a stale passage and be wrong, and it can be correct from the model's own knowledge while ignoring the passages. Grade both, and read the pairs: grounded and wrong means the index needs updating; ungrounded and right means the model is answering from memory and will not survive a topic it does not know.
Answer relevance asks whether the answer addresses the question at all, usually by a judge or by embedding the answer and the question. It catches the long, grounded, correct paragraph that never answers what was asked.
Agent metrics
An agent can reach the right answer by a wrong route: an extra tool call that moves money, a calculation done in its head instead of with the calculator, a loop that stops on a budget. The final answer grade misses all of it. Trajectory precision and recall compare the tools called against an expected plan, with alternative plans labelled where more than one is valid. Argument accuracy checks that the calls carried the right values, which is where the most convincing failures hide: the right tool, the wrong order id. Step count, tool-call count and budget stops are cheap operational signals that catch runaways before the bill does.
Judge metrics
A judge model is a measuring instrument, and instruments are calibrated. Collect human labels on a few dozen cases, run the judge on the same cases, and report accuracy and Cohen's kappa, which corrects for agreement by chance; for numeric scores report mean absolute error. For pairwise judges, run every comparison in both orders and report the position-swap consistency, because judges prefer the first answer, the longer answer and answers in their own style. A judge with kappa below about 0.6 against your labels is telling you about itself.
Operational metrics
Every quality gate should sit next to p95 latency per stage, tokens per request and cost per request, rolled up per feature and per user. A change that improves faithfulness by shifting every question to a larger model has not improved the system until the cost column says so. Sample traces so that every error and every slow request is kept, or the tail you need to see is exactly the one that gets dropped.
From metrics to a decision
Metrics become useful when they gate a change. A regression gate compares a candidate configuration's report to the baseline's and fails if any gated metric drops by more than a tolerance, and, separately, if any protected question that passed before fails now. The per-question rule matters more than it looks: the mean can improve while the most common support question breaks, and only the rows can say so. A gate that exits non-zero is a CI job, and a CI job is the difference between an evaluation someone ran once and an evaluation that runs on every change.