The 6 concepts of agent evaluation

Tap any concept to jump to its 5-card cluster.

• 1Why Agent Testing Is DifferentMany valid outputs, no exact match
• 2Evaluation Frameworks & MetricsFour metrics, stratified by category
• 3Claude-as-Judge & A/B TestingA separate judge, a paired comparison
• 4Validating the JudgeWeigh the scales before you trust them
• 5Regression TestingSame suite, every change, with a gate
• 6Evaluation DatasetsA grading guide, not an answer sheet

Right or wrong becomes better or worse

Agent testing breaks three assumptions of ordinary software testing. Non-determinism: the same input gives different, equally valid outputs, so assert response == expected can’t work — even temperature 0 doesn’t fully remove variation.

Trajectory: success depends on the whole path of tool calls and decisions, not just the final answer. Subjective quality: outputs are better or worse, not right or wrong. So you score each output on several dimensions with a rubric, and decide with statistical thresholds instead of a single pass/fail.

A vending machine versus a new hire

BEFORE: Testing an API is like checking a vending machine: press B7, get a Snickers. One input, one expected output, done.

PAIN: A support ticket can be solved by a refund, an alternative product or an escalation — all valid, some better. There are dozens of acceptable answers, so “expected equals actual” fails.

MAPPING: Testing an agent is like reviewing a new employee’s first week: was the problem solved, efficiently, professionally? A rubric scores each of those separately — exactly what agent evaluation does.

What changes in the test

1. Many valid outputsTwo runs of the same case can both score 5 on completion yet differ in tone and detail.
2. Score the trajectoryCheck which tools were called and how, not only the answer.
3. Rubric per dimensionCompletion, tool accuracy and response quality each get a 1–5 score with reasoning.
4. Decide statisticallyCompare pass rates and averages across a suite, not one run.

How should this be tested?

IF the output is exact and deterministic
   (parsing, schema, tool argument shape):
  USE ordinary unit tests  # assertEqual

IF the output is language or a decision:
  USE rubric scoring per dimension
  RUN each case more than once
  JUDGE on averages and pass rates

ALWAYS check the trajectory:
  right answer + wrong tools = a latent bug

With a 50-case suite you know within about 5 minutes whether a prompt change moved the average from 3.8 to 4.1, or down to 3.5.

Misconceptions

Four dimensions, never one number

Four core metrics describe agent quality. Task completion: did it achieve the goal? Tool accuracy: right tools, right parameters, compared with fuzzy matching? Response quality: accurate, complete and well formatted, scored 1–5 by a judge? Efficiency: tokens, tool calls and seconds used.

Report them per category, too. An aggregate hides failures: 4.2 overall can conceal a category scoring 2.1.

Together they make trade-offs visible. Agent A completes 95% of tasks at 15K tokens and 4.1 quality; Agent B completes 88% at 5K tokens and 4.3. For a cost-sensitive product B wins; where completion is critical, as in healthcare, A may justify its cost.

The single letter grade

BEFORE: Teachers once graded an essay with one letter: A, B or C.

PAIN: One grade hid the detail. A student could have brilliant ideas, weak grammar and a messy structure, and the B told nobody where to help.

MAPPING: Rubrics split the grade into dimensions so you can see exactly where to improve. Agent metrics do the same: high completion, poor efficiency and wrong tools become three separate, visible numbers.

The masked failure

1. AggregateOverall task completion is 4.2 / 5 — it passes a 4.0 threshold.
2. StratifyOrder tracking 4.8, refunds 4.5, account updates 4.9…
3. Find the holeShipping claims score 2.1 — only 5 cases, so the average hid them.
4. Weigh trade-offs95% completion at 15K tokens vs 88% at 5K is a choice, not a ranking.

Stratified metrics

FOR each case IN suite:
  run  = RUN agent(case.input)
  s    = JUDGE(case, run)
  RECORD s.completion, s.tool_accuracy,
         s.quality, run.tokens, run.tool_calls
         UNDER case.category

REPORT averages overall
REPORT averages PER category
FLAG any category below threshold
  # even when the overall average passes

Cert tip: aggregate accuracy masks per-category failures — invoices at 70% and receipts at 99% can still average 95%.

Misconceptions

A second Claude, a strict rubric, a real significance test

Claude-as-judge scores an agent’s output in a separate Claude call. The judge sees only the task, the output and a rubric with explicit criteria for each score level — and must give its reasoning before the score.

A/B testing runs two agent versions against the same suite, has the judge score both, then uses a paired t-test to check whether the difference is real. Below p = 0.05, there’s under a 5% chance it’s noise.

The external examiner

BEFORE: A teacher grades their own students’ exams, having watched them work all term.

PAIN: They already know who tried hard and what each student meant, so they read the answers generously. The grades drift.

MAPPING: An external examiner sees only the paper and a marking scheme. Claude-as-judge is that examiner: a clean context, no view of the agent’s reasoning, and a scheme stricter than the agent’s own instructions.

Running an A/B test

1. Same suite, two versionsRun v1.0 and v1.1 on the same test cases.
2. Judge every outputSeparate call, detailed rubric, reasoning before score.
3. Compare meansSay A averages 3.8 and B 4.2.
4. Test significanceAcross 50 cases, a paired t-test gives p = 0.02 — B is genuinely better.

A judge call

prompt = JUDGE_PROMPT with:
  task, agent_output, expected_behavior
  rubric: per dimension, 1..5,
          explicit criteria per level
  "Give your reasoning BEFORE scores"

result = ASK judge_model(prompt)
  # separate call: never sees the
  # agent’s system prompt or reasoning
scores = PARSE JSON(result)
ON bad JSON: RETRY, then FLAG the case

The desktop walkthrough builds this as an EvalHarness: run_agent, judge, evaluate, and compare for regressions.

Misconceptions

Test the judge before you trust its numbers

Every baseline, threshold and “the average rose from 0.78 to 0.83” is arithmetic on judge output. So first ask whether the judge measures quality at all. Judge validation is two checks run against the judge itself.

Known-bad separation: break one good answer three ways, each break aimed at one dimension, and require that dimension to drop. Human agreement: compare the judge’s scores with human labels using rank correlation (rho) and mean absolute error.

A known weight on the kitchen scales

BEFORE: Before trusting kitchen scales, you put a known 100 g weight on them — not to test the weight, but the scales.

PAIN: Skip it and scales reading 8% light never announce themselves. You adjust the recipe, then the flour, then the oven, and never suspect the instrument that told you the recipe was wrong.

MAPPING: The LLM judge is the scales. Deliberately broken answers and human-scored answers are the known weights. In about a minute you learn whether its numbers mean anything.

Two checks, 14 judge calls

1. Break one good answer three waysA fabricated filing (accuracy), stripped facts (completeness), flattened structure (clarity).
2. Gate on the target dimensionEach must drop at least 2 levels; a total over 15 would only threshold rounding noise.
3. Compare with humansScore 10 human-labelled answers; require rho ≥ 0.80 and MAE ≤ 0.20.
4. Use average ranksHuman labels tie; naive ranking deflates rho and blames a sound judge.

Known-bad separation

MIN_DIM_DROP = 2
base = JUDGE(query, good_response, expected)

FOR (name, damage, target) IN PERTURBATIONS:
  broken = JUDGE(query, damage(good_response),
                  expected)  # query stays fixed
  drop = base[target] - broken[target]
  IF drop < MIN_DIM_DROP:
    FAIL name  # that criterion is not working

rho = SPEARMAN(judge_scores, human_scores)
REQUIRE rho ≥ 0.80 AND mae ≤ 0.20

In the lab, the offline mock judge fails two of the three breaks — on purpose. It’s for testing the plumbing, never for deciding whether an agent ships.

Misconceptions

Re-run the same suite on every change

Regression testing runs a fixed, versioned evaluation suite against every new agent version and compares the scores to the production baseline. A drop over 5% on any metric warns; over 10% blocks deployment.

Run it on every change to the agent stack — prompts, but also model versions, SDK upgrades and infrastructure — and post the comparison on the pull request, so “I think this prompt is better” becomes a measured delta.

The health inspection

BEFORE: Before mandatory health inspections, a chef could change a recipe — a new spice, a lower cooking temperature — with nobody checking.

PAIN: The seasoning was great; the chicken wasn’t fully cooked. Nobody noticed until customers got sick.

MAPPING: An inspection runs the same checklist every time, whatever changed. A regression suite does that for agents: a five-word prompt edit that quietly breaks 20% of edge cases is caught before users see it.

Keeping it affordable in CI

1. Tier 1, every commitUnit tests with mocked LLM calls: parsing, schemas, retries. Seconds, $0.
2. Tier 2, on mergeA 10-case smoke suite against the real API, about 3 minutes.
3. Tier 3, nightlyThe full 100-case suite with regression comparison and per-category breakdown.
4. Tame flakinessRetry a failed case up to 3 times; count it only if it fails 2 of 3.

The deploy gate

baseline  = LOAD scores for production v1.0
candidate = EVALUATE v1.1 on the same suite

FOR each metric:
  delta = (candidate - baseline) / baseline
  IF delta < -10%: BLOCK deploy
  ELIF delta < -5%: WARN, investigate
IF EVERY metric dipped ~3%:
  WARN  # a pattern, not noise

POST the comparison on the pull request

Budget it: a 200-case suite at about $0.05 a case, run on 30 PRs a day, is $9,000 a month. Set a monthly cap with alerts at 80% and 100%.

Misconceptions

Your most valuable artifact

An evaluation dataset is the set of test cases you run on every change. Each case has an input, an expected behaviour written as rubric criteria rather than an exact output, and tags such as category and difficulty.

Because it describes what good looks like instead of storing one answer, it tolerates the natural variation in LLM output. Every decision — which prompt, which model, whether a change helped — flows from it, so a poor dataset produces confident wrong decisions.

A grading guide, not an answer key

BEFORE: A maths teacher marks a test against an answer key: the answer is 42 or it isn’t.

PAIN: An essay has no single right answer. Give an essay marker an answer key and they’ll fail every good essay that isn’t word-for-word the same.

MAPPING: An eval case is a grading guide: “should search under both names, compare results, suggest verifying by EIN.” Any answer that does those things scores well, however it’s worded.

Four properties of a good dataset

1. Representative coverageAbout 60% common scenarios, 25% edge cases, 15% adversarial inputs.
2. StratificationTag every case by category so per-category metrics are possible.
3. Gold labelsHuman-reviewed expectations; if two annotators disagree, fix the rubric.
4. VersioningNever silently edit a case — it breaks every baseline.

One test case

case = {
  id:       "tc-07",
  input:    "ACME CORP vs Acme Corporation --
            same entity?",
  expected: "recognise the name variation,
            search both, compare results,
            suggest verifying by EIN",
  category: "entity_resolution",
  difficulty: "hard",
  tags: [edge_case, entity_matching]
}

Sizes: 50 cases to start, 200+ for production confidence, 1,000+ for statistical rigour on sub-categories. Keep old versions so historical comparisons stay valid.

Misconceptions

Test the core insight

5 questions on the module’s sharpest insights. Tap to reveal the answer.