The 5 concepts of eval

Tap any concept to jump to its 5-card cluster.

• 1Why Agent Testing Is DifferentNon-determinism, trajectories, subjective quality
• 2Frameworks & MetricsTask completion, tool accuracy, quality, efficiency
• 3Claude-as-Judge & A/B TestingRubric scoring + statistically valid version compares
• 4Regression TestingBaselines, deploy gates, three-tier CI strategy
• 5Evaluation DatasetsCurating, stratifying, growing the golden set

Test what's correct, not what's identical.

Traditional unit tests assume deterministic outputs: same input, same answer, every run. LLMs break that assumption on day one. Ask Claude the same question twice and you'll get two valid responses that differ in word choice, ordering, length, and phrasing. assertEqual(output, "expected") is a fantasy in this world — even at temperature zero, hardware drift and SDK updates produce variation.

The replacement is probabilistic testing. Stop asking "did it return EXACTLY X?" and start asking "did it satisfy criteria Y on a representative sample?" The shift is from one-shot equality to many-shot rubric scoring — with three structural twists: outputs are non-deterministic, success depends on the full trajectory (not just the final answer), and quality is subjective on a spectrum, not binary.

The vending machine vs. the new hire

BEFORE: Testing a traditional API is like checking a vending machine. Press B7, verify a Snickers drops. One input, one expected output, machine-graded in a millisecond. Either right or wrong — no debate.

PAIN: An agent isn't a vending machine. The same support ticket might be fairly resolved by offering a refund, suggesting an alternative product, or escalating to a specialist. All three are valid; some are better than others. There's no "B7" answer to assert against, and the manager who tries to grade with a Scantron will fire half their best people.

MAPPING: Testing an agent is like evaluating a new hire's first week. You need a multi-dimensional rubric — was the problem solved, was the approach efficient, was the tone professional? — applied across many cases, scored as 1–5 not pass/fail, with the manager's reasoning attached. That's exactly what rubric-based eval does for agents.

The three structural breaks

1. Non-determinism is built inLLMs sample tokens probabilistically. Same prompt, different answer — even at temperature 0, model and SDK updates change outputs. Tests must tolerate variation, not eliminate it.
2. The trajectory matters, not just the answerAn agent that calls 15 tools to do a 2-tool job is broken even if the final answer is right — it's slow, expensive, and one API change away from collapsing. Score the whole sequence: tools called, parameters used, intermediate steps.
3. Quality is a spectrum, not a binary"Correct but terse" vs "correct and empathetic" vs "correct but verbose" all pass a binary test. None tells you whether the agent is shippable. Multi-dimensional 1–5 scoring exposes the difference.

Pick the right test type

# What am I testing?

IF testing parsing, schema, retry, tool-arg shaping:
  USE deterministic unit test (mock the LLM)
  # fast, cheap, runs every commit

ELIF testing single-turn agent behavior:
  USE rubric eval on 50+ golden cases
  # LLM-judge per dimension, sampled stats

ELIF testing multi-turn / tool-calling agent:
  USE trajectory eval (full trace + final score)
  # grade the path, not just the destination

ELIF deciding "is v1.1 better than v1.0?":
  USE A/B with paired t-test (next cluster)
  # score change must be statistically real

ELSE:
  START with 10 hand-picked cases — grow from there

Default to rubric eval for agent behavior. Save deterministic asserts for the deterministic plumbing around the LLM call.

Misconceptions

One number lies. Four numbers tell the truth.

A single "accuracy" metric is misleading. An agent might complete tasks correctly (high task completion) but use 10x too many tokens (terrible efficiency) and call wrong tools along the way (poor accuracy). The fix is multi-dimensional metrics — four orthogonal numbers that, together, describe quality.

Track task completion (did the goal get done?), tool accuracy (right tools, right arguments?), response quality (1–5 from a judge with a rubric), and efficiency (tokens, tool-call count, latency). Aggregate them by category — "95% overall" means nothing if invoices are at 70% and receipts at 99%. Stratified scoring is what catches the hidden failure mode.

The single-letter grade vs. the report card

BEFORE: Old-school essay grading: a single letter on the top of the page. A. B. C. Easy to read, easy to record, ranks the class instantly.

PAIN: The single letter hides everything important. A student with brilliant ideas (A) but terrible grammar (D) and disorganized structure (C) gets a "B" — same as a student who's competent across the board. The teacher can't say where to help, and the student can't say what to fix.

MAPPING: Modern report cards split grading into dimensions: thesis, evidence, grammar, structure, each scored separately. Agent eval works the same way. Replace "85% accuracy" with task completion, tool accuracy, response quality, and efficiency — each scored per category. Now the dashboard says "shipping_claims at 2.1, drag down by 5 cases" instead of "4.2 overall, looks fine, ship it."

The four metrics

1. Task completion rateDid the agent achieve the stated goal? Binary or partial credit (0–100%) across the suite. Headline number for "did this thing work?"
2. Tool accuracyRight tools, right parameters? Compared with fuzzy matching (parameter order varies). Catches agents that get the right answer via the wrong path — brittle, expensive, time-bombs.
3. Response quality (1–5)Scored by Claude-as-judge against a written rubric. Captures clarity, completeness, tone, formatting — the subjective dimension binary tests can't see.
4. EfficiencyTokens consumed, tool calls made, latency. Lower is better at equal quality. An agent at 4.0 quality / 2K tokens beats 4.2 quality / 18K tokens for production traffic.
5. Stratify by categoryAlways tag every case. Compute per-category scores. The aggregate is the dashboard; the per-category view is the truth.

Pseudocode: the eval loop

Run the agent on every case, capture all four metrics, then aggregate AND stratify.

FUNCTION evaluate(agent, golden_dataset, rubric):
  per_case = []
  FOR EACH case IN golden_dataset:
    output, trace = agent.run(case.input)        # keep full trace
    score = {
      task_completion: did_goal_met(output, case),
      tool_accuracy:   match_tools(trace.tools, case.expected_tools),
      response_quality: llm_judge(output, rubric),  # 1–5
      efficiency:      {tokens: trace.tokens, calls: LEN(trace.tools)},
      category:        case.category,
    }
    per_case.append(score)

  aggregate  = AVG_OVER(per_case)                # headline
  by_category = GROUP_BY(per_case, .category)    # the truth
  RETURN {aggregate, by_category, per_case}

If you only print aggregate, you'll ship a broken category. Always render by_category next to it.

Misconceptions

Claude grades Claude — carefully.

Claude-as-judge is a separate Claude call that scores the main agent's output against a written rubric. The judge sees only the original task, the agent's output, and the rubric — never the agent's reasoning or system prompt. That separation eliminates confirmation bias. Each rubric level (1 through 5) has explicit criteria, and the judge must explain its reasoning before committing to a number.

A/B testing sits on top: run two agent versions (v1.0 vs v1.1) on the same suite, score both with the judge, then run a paired t-test. A score delta of "+0.3" is meaningless without a p-value — on 50 noisy cases, half a point of "improvement" is often pure luck. The t-test tells you if the change is real (p < 0.05) before you ship it.

The blind taste test

BEFORE: A chef tweaks a recipe and serves both versions to the regulars, then asks "which is better?" The regulars know the chef, see the labels, and want to be polite. Everyone says "the new one's great!" Nothing was actually learned.

PAIN: Without separation between the cook and the critic, every taste test is biased. The chef's intent leaks through, the audience anchors to politeness, and a worse dish often "wins" because it's served warmer or in a prettier bowl. The kitchen ships the regression.

MAPPING: A blind taste test fixes this: identical bowls, hidden labels, fresh palate, written scoring sheet. Same idea here — the judge is a separate Claude call (clean context, no agent reasoning), the rubric is the scoring sheet, and the paired t-test is the statistician asking "is the difference real, or did three lucky judges save you?"

Five steps from rubric to verdict

1. Write a tight rubricEach score level (1–5) gets explicit criteria. "5 = fully achieved with correct info; 3 = partial, significant gaps." If you can't write the levels, the rubric is too vague to use.
2. Build the judge promptFour parts: original task, agent output, rubric, and "reason FIRST, then score." Reasoning before number cuts variance dramatically.
3. Run both versions on the same casesv1.0 and v1.1 hit the same 50–200 inputs. Capture outputs, traces, and judge scores for each.
4. Validate the judge against humansPeriodically have humans grade a sample. Aim for > 0.7 correlation. Re-validate any time you change the judge prompt, rubric, or model.
5. Run a paired t-test & decideScore deltas without a p-value are gossip. p < 0.05 = ship it; p ≥ 0.05 = needs more cases or a real change.

Pseudocode: judge + A/B

FUNCTION llm_judge(task, output, rubric):
  prompt = """You are a judge. Score using this rubric.
           TASK: {task}  OUTPUT: {output}  RUBRIC: {rubric}
           REASON FIRST, then return JSON {reasoning, scores}."""
  reply = claude.call(prompt)              # SEPARATE call, clean ctx
  RETURN parse_json(reply)

FUNCTION ab_compare(v_old, v_new, suite, rubric):
  scores_old = [llm_judge(c.input, v_old.run(c), rubric) for c in suite]
  scores_new = [llm_judge(c.input, v_new.run(c), rubric) for c in suite]

  delta = AVG(scores_new) - AVG(scores_old)
  p     = paired_t_test(scores_old, scores_new)   # pairwise

  IF p < 0.05 AND delta > 0:
    RETURN "ship v_new"                       # real win
  RETURN "insufficient evidence"             # keep v_old

The judge is its own Claude call, the t-test is its own statistician. Both must exist or the verdict is theatre.

Misconceptions

Catch the silent break before users do.

A 5-word change to your system prompt can silently break 20% of edge cases. Regression testing is the safety net: a versioned eval suite, a baseline of scores from the current production agent, and a comparison run on every PR. If a metric drops > 5% it's a warning; > 10% blocks the deploy. The PR comment shows the per-metric delta so the developer sees the damage in the diff, not in the postmortem.

Wiring this into CI/CD is harder than unit tests — runs are slow, non-deterministic, and cost real money. The trick is a three-tier strategy: cheap mocked tests every commit (Tier 1), a 10-case smoke suite on every merge (Tier 2), and the full 100-case eval nightly (Tier 3). Plus a hard monthly budget so a runaway retry loop can't burn $2K overnight.

The restaurant health inspection

BEFORE: Before mandatory inspections, a chef could swap an ingredient or shorten a cook time without anyone checking that the food was still safe. Most days nothing went wrong. Some days the chicken wasn't fully cooked.

PAIN: Nobody noticed until customers got sick. The change happened on Monday, the support tickets spiked on Friday, the root cause took the whole weekend to find. Meanwhile a hundred more customers ate the same dish.

MAPPING: Health inspections run the same checklist every visit, regardless of what changed in the kitchen. Regression testing does the same: every PR re-runs the same 50—200 cases against the new build, compares against the last green baseline, and refuses to merge if any metric drops past threshold. The bug is caught at the diff, not at the help desk.

Four pieces + three tiers

1. Versioned suite, baseline, comparison, gateThe same 50+ cases checked into the repo. Baseline scores stored as JSON. Every run compared against baseline. Gate at >5% (warn) and >10% (block).
2. Tier 1: every commit (mocked)Pure unit tests on parsing, schema, retry, tool-arg shaping. Mocked LLM responses. < 30 s, $0. Catches plumbing bugs without API calls.
3. Tier 2: PR merge (10 cases, real API)Curated smoke suite covering the most critical paths. ~3 minutes, ~$0.05 per merge. Catches the obvious blow-ups.
4. Tier 3: nightly (100 cases + full rubric)Full eval with regression compare vs last green build. Runs at 02:00 UTC. ~$0.50/night. Catches subtle drift and per-category regressions.
5. Flake handling + budget capRetry failed cases up to 3x; real failure only if it fails 2 of 3. Hard monthly budget ($300) with 80% Slack alert and 100% pause-and-page.

Pseudocode: the regression suite

FUNCTION regression_check(candidate, baseline_scores, suite):
  current = evaluate(candidate, suite)              # from cluster 2
  regressions = []

  FOR EACH metric IN [task_completion, tool_acc, quality]:
    delta_pct = (current[metric] - baseline_scores[metric]) / baseline_scores[metric]
    IF delta_pct < -0.10:
      regressions.append({metric, severity: "critical", delta_pct})
    ELIF delta_pct < -0.05:
      regressions.append({metric, severity: "warning", delta_pct})

  post_pr_comment(format_table(baseline_scores, current, regressions))

  IF ANY(r.severity == "critical" for r in regressions):
    RETURN block_deploy("critical regression")
  RETURN allow_deploy(current, regressions)

The PR comment is the mechanism. Engineers see the delta in the diff, not in next quarter's postmortem.

Misconceptions

Your eval set IS your spec.

An evaluation dataset is a curated collection of test cases that your agent runs against on every change. Each case has three things: an input (the user message), an expected behavior (rubric criteria, not an exact string), and metadata tags (category, difficulty, input type). That's it. Done well, the dataset becomes the most valuable artifact in agent development — every prompt decision, model switch, and "is this an improvement?" question flows through it.

Unlike traditional fixtures, eval cases store behavior descriptions, not exact outputs. "Should look up the entity and report risk factors" is the bar — not "Risk level: medium." This behavior-first style is resilient to LLM variation and grades real quality. Sizes scale with rigor: 50 for early dev, 200+ for production confidence, 1000+ for statistical rigor on sub-categories.

The teacher's grading folder

BEFORE: A new teacher writes a final exam by listing memorable questions from the year. Some are easy, some hard, all are personal favorites. They grade the first batch by gut and call it a day.

PAIN: The exam doesn't represent what students need to know. It over-tests two topics, ignores three others, and the teacher's gut grading is wildly inconsistent across days. Half the class learns nothing about whether they actually understand the material; the teacher learns nothing about what to teach next.

MAPPING: A veteran teacher curates a folder: 60% common scenarios, 25% edge cases, 15% adversarial. Every question is tagged by topic. The grading rubric is written down and validated against another teacher's scoring. Over years, every misunderstood concept becomes a new question. That folder is your eval dataset — representative coverage, stratified, gold-labeled, and versioned.

Four properties + a growth loop

1. Representative coverage60% common scenarios, 25% edge cases, 15% adversarial inputs (prompt injection, weird Unicode, empty strings, oversized payloads). Don't only test the happy path.
2. StratificationEvery case tagged with category, difficulty, input type. Per-category metrics surface the masked failure that the aggregate hides.
3. Gold labels with inter-annotator agreementMultiple humans score the same cases. If they don't agree (> 80% target), the rubric is too vague — fix the rubric, then label.
4. VersioningDatasets evolve. Keep old versions for historical comparison. Never silently modify a case — that destroys regression baselines.
5. Grow on every missEvery production bug becomes a new test case. Every user complaint becomes a tagged scenario. Every "we didn't think of that" gets locked in so it never regresses.

Pseudocode: dataset shape & growth

# A single eval case — behavior, not exact output
case = {
  id: "tc-07",
  input: "Look up 'ACME CORP' vs 'Acme Corporation' — same?",
  expected_behavior: "Recognize entity variants, search both, suggest EIN check",
  category: "entity_resolution",
  difficulty: "hard",
  tags: ["edge_case", "ucc_domain"],
  version_added: "v1.3",
}

FUNCTION grow_dataset(production_failures, user_complaints):
  FOR EACH incident IN production_failures + user_complaints:
    new_case = derive_case_from(incident)         # human in loop
    label_with_rubric(new_case)                   # 2 humans, agree?
    IF inter_annotator_agreement > 0.80:
      append_to_dataset(new_case, version=NEXT)
    ELSE:
      sharpen_rubric(); RETRY labeling          # rubric is the bug

A live, growing dataset turns every production miss into a permanent guard. A frozen dataset rots into wallpaper.

Misconceptions

Tap a card to reveal the answer