CCAR-P Evaluation Strategy: Eval Datasets, Metrics & Diagnosis [2026]

Domain 4 of the Claude Certified Architect - Professional (CCAR-P) exam, Evaluation, Testing & Optimization, carries 16% of the blueprint, which works out to roughly ten of the 63 scored questions. It is also the domain where strong builders lose points, because the questions ask you to prove that a system works, and proof requires a different discipline from building. This guide covers the evaluation strategy the exam expects an architect to carry: choosing the metric that matches the failure you fear, assembling evaluation datasets that catch regressions before customers do, combining code graders, model graders, and human review, wiring regression gates into CI, and diagnosing whether a failing system has a prompt problem, a retrieval problem, or the wrong model tier. Two worked scenarios show how these ideas resolve into one exam-correct answer.

How the exam tests Domain 4

Domain 4 questions almost always start in production: a system is live, someone has noticed a symptom (wrong answers, a blown latency budget, a cost overrun, an injection that got through), and the stem asks what the architect should measure, build, or change first. Four question shapes recur: pick the metric that matches a stated business failure; identify what is missing from an evaluation approach (usually a held-out set, per-slice reporting, or a human calibration step); attribute a failure to prompt, retrieval, or model tier from a set of symptoms; and choose the first optimization lever that fits a cost or latency constraint without breaching a quality floor.

The distractors are real techniques applied at the wrong stage. Upgrading the model tier when retrieval never surfaced the answer, A/B testing before any offline evaluation exists, or reporting one aggregate accuracy number when the scenario describes several customer segments are all recognizable traps. Anthropic's guidance on defining success criteria frames a good target as specific, measurable, achievable, and relevant, and stresses that most real systems need several criteria at once. Holding that multidimensional frame eliminates single-metric answers quickly.

Metric selection: match the metric to the failure

The blueprint names five metric families: accuracy, latency, cost, safety, and security. Each family answers a different question, and each has a preferred measurement.

Accuracy and factuality. For closed tasks such as classification, routing, or field extraction, accuracy is a labeled-set comparison: exact match, or precision and recall combined into an F1 score (the harmonic mean of the two, so a system cannot look good by being merely permissive or merely strict). For open-ended generation and retrieval-augmented generation (RAG, where the model answers from documents retrieved at request time), the useful measure is factuality or faithfulness: does the answer stay inside the evidence it was given? A model grader with a rubric, calibrated against human labels, usually scores it.

Latency. Averages hide the requests that breach an SLA (service-level agreement, the response-time commitment made to the business). Report percentiles: p95 is the latency under which 95% of requests complete, and p99 is the same threshold for 99% of requests. For streaming interfaces, add time to first token, the delay before the first piece of output appears, since users judge responsiveness by that moment.

Cost. Cost per request is the easy number, and cost per successful task is the honest one, because it divides total spend (including retries, tool round-trips, and calls that produced unusable output) by the tasks that actually met the quality bar. Track input and output tokens separately, plus cache hit rate when prompt caching (reusing a stored, stable prompt prefix so repeated tokens are billed at a discount) is part of the design.

Safety violation rate. The share of outputs on a deliberately hostile test set that a classifier, a model grader, or a human reviewer flags as harmful, off-policy, or outside the assistant's scope. The official docs illustrate a measurable safety target as a bounded fraction of flagged outputs across many trials: a rate with a denominator, on a set you control.

Security test pass rate. The share of a red-team suite that the system resists: prompt injection through user input and through retrieved documents, attempts to exfiltrate data through tool calls or logs, and attempts to trigger tools outside the user's authorization. Graders here are mostly code assertions, since "did the agent call the payments tool" is a yes-or-no fact.

One more family appears in agent scenarios: consistency. Anthropic's engineering guidance on agent evals distinguishes pass@k, the chance that at least one of k trials succeeds, from pass^k, the chance that all k trials succeed; a customer-facing agent that must work every time is judged on pass^k.

Read every stem for the failure the stakeholder actually named and pick the metric family that measures it. When a scenario mentions an SLA and a compliance regime together, a "Select TWO" question is often asking for a latency percentile plus a safety or security rate.

Building evaluation datasets

A metric is only as good as its dataset. Four dataset ideas carry most Domain 4 questions.

Golden sets

A golden set is a curated collection of inputs paired with known-correct outputs or verifiable outcomes, drawn from real traffic so its distribution mirrors production. For a support assistant that means real tickets with the correct routing label; for a RAG system, real questions with the passages that answer them. Anthropic's guidance is blunt: more cases with automated grading beat fewer hand-graded ones, because volume surfaces the rare failures.

Adversarial sets

An adversarial set holds the cases built to break the system: injection strings hidden in user text and in retrieved documents, out-of-scope requests, ambiguous questions with two defensible answers, inputs long enough to crowd the context window, malformed or empty inputs, and questions the corpus cannot answer (so the correct behavior is to say so). The official develop-tests page lists irrelevant or nonexistent input data, overly long input, poor or harmful user input, and ambiguous cases as the edge cases every eval should include. Keep adversarial cases in known proportions so a per-slice safety rate means something.

Stratified sampling

Aggregate accuracy hides the slice that is failing. Stratified sampling means defining the slices that matter to the business (intent type, document family, language, customer tier, difficulty band, region) and sampling enough cases from each that its score is meaningful on its own, then reporting per slice. A scenario in which overall accuracy looks healthy while one customer segment keeps complaining is a stratification question, and the answer is a per-segment breakdown before any prompt change.

Size, growth, and hold-out

Anthropic's engineering team recommends starting with 20 to 50 simple tasks drawn from real failures and growing from there; every production incident becomes a new case. Two disciplines protect the numbers. First, hold the sign-off set out of prompt development, because tuning prompts against the cases you report on inflates the score until production deflates it; keep a development set for iteration and a held-out set for release decisions. Second, refresh the set as traffic drifts and retire saturated cases, since a suite sitting at 100% says nothing about the next regression. Write tasks unambiguously enough that two domain experts would reach the same verdict, and balance positive and negative cases so the system cannot pass by refusing everything.

Mixed methodology: code graders, model graders, and human review

The blueprint says "mixed-methodology test frameworks" because no single grader covers all five metric families.

Code-based graders own everything objective: valid JSON with three required fields, an agent that must never call the refund tool without an approval token, a p95 latency budget. Write an assertion. LLM-as-judge (a separate model call that scores an output against a written rubric, either as a pass/fail or on a short scale) owns qualitative properties. Two rules from the official docs matter on the exam: use a different model to grade than the one that produced the output, and calibrate the judge by periodically sampling its scores against human ratings, then rewrite the rubric when they diverge. Human review then concentrates on the high-stakes slice and on calibration, where its cost is justified.

Anthropic's agent-eval guidance adds two habits. Prefer outcome-based grading (did the environment end in the right state) over rigid step-sequence checks that punish valid alternative paths, and read transcripts regularly, because grader bugs can make a capable system look broken. Their published example of a score jumping after a numeric-formatting grader was fixed is a reminder that the eval is code too, and it can be wrong.

Regression gates in CI

Anthropic's engineering post distinguishes capability evals, which start at low pass rates and guide improvement, from regression evals, which sit near 100% and exist to catch quality loss. A regression gate is the second kind wired into continuous integration (CI, the automated pipeline that runs tests on every change): whenever a prompt, model version, retrieval configuration, tool definition, or Skill changes, the suite runs and the change is blocked if any slice drops below its floor or if latency and cost per task exceed their budgets.

Three details make the gate work. Prompts and rubrics are versioned in source control alongside the code, so every eval result is attributable to a change. Because model output varies, must-work cases run several trials and are gated on pass^k so a single lucky pass cannot clear them. And the suite is split by runtime: a fast regression set runs on every change, while the larger capability set runs on a schedule, and saturated capability cases graduate into the regression suite as harder cases replace them.

The exam version is usually a scenario in which a prompt change improved one metric and silently degraded another; the missing practice is a regression suite that reports every metric family per slice on every change.

A/B testing and shadow testing

Offline evals prove a candidate works on the cases you thought of. Two live-traffic methods prove it works on the cases you did not.

Shadow testing runs the candidate in parallel with production on real requests without serving its output to anyone. Because users never see it, it carries no user risk, and because it sees the real distribution, it produces trustworthy p95 latency, cost per task, and (with a model judge comparing candidate output against production output) accuracy numbers before any rollout decision.

A/B testing splits live traffic between the control and the variant and compares an outcome users experience: resolution rate, escalation rate, satisfaction, task completion. The discipline the exam looks for: change one variable per experiment (prompt version, model tier, or retrieval setting, never several at once), predefine the success metric and the sample size before starting, hold until the result is statistically meaningful, and roll out on evidence. Picking the variant that felt better after ten spot checks is the anti-pattern.

The order matters: offline suite, then shadow, then A/B, then rollout with monitoring. A stem that proposes A/B testing a change that has never been evaluated offline is describing a process gap.

Diagnosing failures: prompt, retrieval, or model mismatch

Given a failing system, the exam wants the root cause, because each cause has a different fix and a wrong attribution costs an engineering cycle. Work the tree in order, with traces (the per-request record of inputs, retrieved context, tool calls, and outputs) as the evidence.

1. Reproduce it. Does the failure appear on the eval set? If it only exists in a user complaint, capture the trace and add the case, because nothing downstream is reliable without a reproducible example.
2. If retrieval is in the loop, inspect what was retrieved. Did the retrieved context contain the answer?
   • No: this is a retrieval failure. Fix chunking (how documents are split; chunks that are too large dilute the signal and chunks that are too small strip context), embedding choice, query rewriting, hybrid keyword-plus-vector search, re-ranking, or index freshness.
   • Yes: move on; the model had the evidence.
3. Is the answer unfaithful to the evidence it had? If the context contained the answer and the model still contradicted or embellished it, the failure is grounding, the practical face of hallucination (the model producing content unsupported by its inputs). Tighten grounding instructions, require citations to retrieved text, instruct the model to say when the answer is absent, and add a faithfulness grader so the regression suite catches it next time.
4. Does the failure hit every input type about equally? Uniform failure across slices points at the prompt: ambiguous instructions, missing decision criteria, no examples for a tricky format, or an output structure the model keeps drifting from. Rewrite the criteria to be explicit and testable, add two to four targeted examples, and enforce structured output where the shape matters.
5. Does it concentrate on the hardest slice? If easy inputs pass and long, multi-step, or ambiguous inputs fail, suspect model mismatch: the selected tier is too small for the task, or the context has grown past what it handles well. The diagnostic is cheap: run the identical prompt and inputs on a higher-capability tier. If the failure disappears, the answer is routing that slice to the stronger model.
6. Is it intermittent on the same input? Same input with different verdicts across repeated trials points at non-determinism or a flaky tool. Measure it with pass^k, tighten the output schema, and check tool reliability before touching the prompt.

The token and latency optimization loop

Optimization questions on CCAR-P are evaluation questions in disguise, because the trap answer is the lever that saves money while quietly lowering quality. The loop that keeps you honest:

1. Baseline everything. On the held-out set, record accuracy per slice, safety and security rates, p95 and p99 latency, tokens per task split into input and output, cache hit rate, and cost per successful task.
2. Pull one lever. Trim context and prune verbose tool outputs; move the stable prompt prefix behind prompt caching; route the easy slice to a faster tier while the hard slice keeps the stronger one; send non-interactive bulk work through the Message Batches API (asynchronous, discounted processing within a completion window, unsuited to interactive latency and well suited to running large evaluations); stream responses to cut time to first token; enforce structured output to remove retry loops.
3. Re-run the regression gate. The quality floor holds per slice, or the change does not ship.
4. Confirm on real traffic. Shadow test for latency and cost, then A/B if the change is user-facing.
5. Log and repeat. Keep the trace for every step; without it you cannot attribute the next improvement or regression to its cause.

The mechanics of each lever, including what prompt caching stores and when model-tier routing pays, are in the CCAR-P cost and latency optimization guide. Where a lever touches a guardrail (trimming a safety preamble to save tokens, say), the governance and guardrails guide covers what must survive the cut.

Worked scenario 1: confident wrong answers from a policy assistant

An insurance company runs a RAG assistant over its underwriting policy library. Support agents report that it answers confidently and wrongly several times a day. The product owner proposes moving to the most capable model tier immediately. The traces show that in most failing cases the retrieved chunks came from the correct policy document but from sections that did not contain the specific clause the question asked about; in a minority of cases the chunks contained the clause and the assistant paraphrased it incorrectly.

Working the tree: the failures reproduce, retrieval is in the loop, and the retrieved context lacked the answer in most cases. That is a retrieval failure first (chunk boundaries and ranking are surfacing the right document at the wrong granularity), with a smaller grounding failure behind it. The immediate work is to build a stratified eval set from the logged failures, split by whether the context contained the answer, then fix chunking and add re-ranking for the first slice, and add grounding instructions plus a faithfulness grader for the second.

The exam-correct choice and why: diagnose from traces and fix the retrieval stage first, measured against a stratified eval set. A model upgrade addresses none of the majority failures, since no model can answer from evidence it never received, and it raises cost per task across all traffic to fix a problem that lives in the pipeline. The distractor "add a stronger disclaimer" changes nothing measurable, and "increase retrieval depth to twenty chunks" trades a ranking symptom for a context-length and cost problem without checking whether ranking is the issue.

Worked scenario 2: a cheaper tier under a strict latency SLA

A ticket-triage pipeline classifies incoming support tickets and routes them to teams. It runs on a mid-tier model, meets the p95 latency SLA agreed with operations, and costs more per month than the budget allows. The team wants to switch to the fastest tier and proposes an A/B test on live traffic next week. There is currently no evaluation suite; quality has been judged by spot checks.

The order of operations is the whole answer. First, build the golden set from labeled historical tickets, stratified by destination team and by ticket difficulty, and record the current tier's per-slice accuracy, p95 latency, and cost per successful task as the baseline. Second, run the candidate tier offline against that set and require the accuracy floor per slice. Third, shadow the candidate on live traffic to measure real p95 latency and cost with zero user exposure. Only if both pass does an A/B test on a behavioral metric such as reroute rate make sense, with one variable, a predefined metric, and a predefined sample size.

The exam-correct choice and why: offline evaluation on a stratified golden set, then a shadow test, before any A/B test. Jumping to A/B exposes users to an unvalidated model and cannot even define success, because there is no baseline. Shadow testing answers the latency and cost claims on the real traffic distribution at no user risk, which is exactly what a tier swap needs proven. The tempting distractor, "route all tickets to the fast tier and monitor complaints," measures quality through customer harm; the sophisticated-sounding distractor, "add a second model to verify each classification," doubles cost per task in a scenario whose binding constraint was cost.

Frequently asked questions

Next steps. Put the diagnosis tree to work on real stems in the CCAR-P practice questions with explanations, then run a full timed exam on Preporato's CCAR-P practice tests, where six 63-question tests are included with Preporato Pro (see pricing). The CCAR-P cheat sheet condenses the metric table and the symptom-to-cause map for a final-week review.

Sources:

• Define success criteria (Claude Developer Platform docs)
• Create strong empirical evaluations (Claude Developer Platform docs)
• Demystifying evals for AI agents (Anthropic Engineering)
• Batch processing (Claude Developer Platform docs)
• Claude Certified Architect - Professional (Anthropic Partner Academy)