The 7 concepts of eval

Tap any concept to jump to its 5-card cluster.

• 1Why Evaluation MattersSilent regressions cost weeks
• 2Three Levels of EvalUnit, integration, LLM-as-judge
• 3Building an Eval DatasetThe golden test set
• 4Unit Testing Your ToolsFast, deterministic, zero LLM
• 5Integration TestingThe full agent loop
• 6LLM-as-JudgeScoring what code can't
• 7Validating the JudgeMeasure the ruler first

"Seems to work" is not a quality standard

You shipped an agent and it looks fine in the demo. But every prompt tweak, model bump, or new tool is a change that can silently degrade quality. Without a testing harness you have no idea whether 50 small "improvements" quietly eroded the agent.

Eval is the automated regression check that catches this. Google's internal research found ~40% of LLM pipeline changes caused a regression in at least one metric — even changes meant as improvements. A 5-minute eval suite on every PR prevents weeks of production debugging.

The bridge inspector who checks once

BEFORE: Imagine a bridge inspector who only checks a bridge the day it's built — never after years of traffic, weather, and stress. Day one it looks perfect, everyone drives across, sign-off complete.

PAIN: A hairline crack starts small and propagates silently for three years. Every load felt fine right up until a truck crosses and the span fails — there was no system to detect incremental degradation.

MAPPING: Your agent is the bridge; every prompt change or model bump is a new load cycle. Evals are the structural inspection program — running the same checks over and over so a slow crack in quality shows up before it collapses in production.

How silent regressions happen

1. A change shipsSomeone edits the system prompt "to improve tone" or bumps the model version.
2. No eval runsThe demo still looks fine, so the change merges with no automated check.
3. Quality drifts unseenThe change subtly shifts behavior — e.g. a false-merge rate climbs to 18% (9,000 bad records/day) with nobody watching.
4. Discovery is expensiveWeeks later an audit finds it; days of debugging trace back to one unmeasured change.

Do you need an eval suite?

IF the agent runs in production
   or touches real data:
  → evals are non-negotiable

IF you change prompt / model / tools:
  → re-run evals BEFORE merging

IF "it looked fine in the demo":
  → NOT evidence — measure it

# ~40% of LLM changes regress
# something. Assume yours might.

Misconceptions

Eval is a layered system, not one test

You test at three levels, each catching a different class of failure. Unit tests check individual tools in isolation. Integration tests run the full agent loop against known inputs. LLM-as-judge evals score subjective qualities — reasoning, faithfulness, relevancy — that no deterministic assertion can express.

You need all three because each has a different cost-to-confidence ratio: units are near-free and run every commit, integration runs every PR, judge tests are expensive and run nightly or on major changes.

Inspecting a car: parts, road test, ride quality

BEFORE: Suppose you judged a whole car with a single test drive around the block and called it "good."

PAIN: A block-long drive misses a worn brake pad (a part fault), a stalling gearbox under load (a system fault), and a jittery, unpleasant ride (a quality fault). One test can't see all three failure types.

MAPPING: So a real inspection uses three layers: bench-test each part (unit tests), a full road test under load (integration), and a rider scoring comfort and smoothness (LLM-as-judge). Each layer catches what the others structurally cannot.

The three levels

1. Unit testsIndividual tool functions in isolation. Deterministic, no LLM, <10ms each, 100+ cases.
2. Integration testsThe full agent loop end-to-end against a known input, with a deterministic pass/fail. ~50 cases, seconds each.
3. LLM-as-judgeA second model scores quality dimensions code can't measure. ~20 cases, ~60s each, expensive.

Which level catches this bug?

IF a single tool returns wrong /
   crashes on an edge case:
  → UNIT test

IF the agent calls wrong tools or
   loops / stops incorrectly:
  → INTEGRATION test

IF reasoning is incoherent or the
   answer drifts / hallucinates:
  → LLM-as-JUDGE

# cheap+frequent at the bottom,
# expensive+rare at the top.

Misconceptions

No golden set, no ruler

An eval harness without a good golden test set is a ruler with no markings. A golden set is a curated collection of (input, expected_output) pairs where the correct answer is known and validated — treated as ground truth.

Each case captures the input query, the expected (ordered) tool calls, and pass criteria like a minimum confidence and required keywords. Storing this as data (JSON), separate from test logic, lets domain experts add cases without touching code.

The teacher's answer key

BEFORE: Picture grading 100 exams with no answer key — just your gut feeling about whether each response "seems right."

PAIN: Your standard drifts by the hour, two graders disagree, and you can't prove a score is correct. There's nothing to measure against.

MAPPING: The golden set is the answer key written by an expert in advance. Now grading is objective and repeatable: compare the agent's answer to the key. Add adversarial and boundary questions to the key so it probes the hard cases, not just the easy ones.

Building the golden set

1. Define a case schemaEach case: an id, the input, expected tool calls, and pass criteria (min/max confidence, keywords, decision).
2. Source from real logsStart with actual production inputs so cases reflect true usage.
3. Add adversarial & boundary casesInclude near-matches that should NOT merge and ambiguous ~0.75-confidence cases.
4. Size it right30+ for integration, 100+ for unit, 20–30 for judge (each judge case costs an LLM call).

Pseudocode

# one golden case (stored as data)
case = {
  id:    "ER-001",
  input: "Are 'Acme LLC' and" +
         "'ACME, L.L.C.' the same?",
  expected_tools: ["search", "fuzzy_match"],
  expect: { decision: "merge",
            min_confidence: 0.90 },
}

FUNCTION load_cases(path):
  data = read_json(path)
  IF data.cases is empty:
    fail("add cases before running")
  RETURN data.cases

Misconceptions

Tools are pure functions — test them like any code

Agent tools receive inputs and return outputs with no LLM involved. That makes them ideal for ordinary unit tests: fast, deterministic, and zero LLM cost. fuzzy_match("Acme LLC","ACME LLC") should score above 0.9; a search for a nonexistent name should return an empty list, not crash.

Parametrize the tests so one table of cases covers many inputs, and one failing case never aborts the rest. A whole tool suite runs in a fraction of a second on every commit.

Bench-testing each component before assembly

BEFORE: Imagine building a machine and only ever turning it on fully assembled to see if it runs.

PAIN: When it fails, you have no idea which of 40 parts is at fault — you tear the whole thing apart to find one bad bearing.

MAPPING: Unit tests bench-test each part on its own before assembly: spin the bearing, load the spring, check the gauge. If a part fails, you know exactly which one and why — in milliseconds, before it ever reaches the full machine.

Unit-testing a tool

1. Pick the tool & inputsList representative inputs plus edge cases: empty string, identical strings, clearly-different pairs.
2. State the expected rangeFor fuzzy match, a min/max score; for search, a min/max result count.
3. Assert — and check for NoneTools return None on not-found, they don't raise. Assert against None explicitly or you miss silent failures.
4. ParametrizeAdd new cases by editing the table, not writing new functions.

Pseudocode

# table of (a, b, min, max) cases
cases = [
  ("Acme LLC", "ACME, L.L.C.", 0.88, 1.0),
  ("",         "Acme LLC",     0.0,  0.1),
  ("Acme",     "Acme",          1.0,  1.0),
]

FOR EACH (a, b, lo, hi) IN cases:
  result = fuzzy_match(a, b)
  assert result is not None
  assert lo <= result.score <= hi

# not-found returns None, never raises
assert search("no_such_name") == []

Misconceptions

Test the reasoning that connects the tools

An agent is more than its tools — it's the reasoning that sequences them. Integration tests run the full agent loop against a known input and verify the output with a deterministic assertion: the LLM is called, but pass/fail is decided by code.

Assert on structure, not text: which tools were called and the numeric output (confidence range, decision). This survives paraphrasing — refactoring the system prompt won't break a well-written integration test.

The full road test under load

BEFORE: Every individual car part passed its bench test, so you assume the assembled car drives fine.

PAIN: Good parts can still combine badly — the transmission shifts at the wrong moment, the car stalls turning uphill. No single part is broken; the coordination is.

MAPPING: The integration road test drives the whole car on a known route and checks the outcome: did it get up the hill, in the right gear, within the time? You don't grade the exact engine noise (the text) — you grade that it did the right things (the tool sequence and result).

One integration case, end to end

1. Run the real agentFeed the golden input; cap iterations and set a timeout so a cold model can't hang CI.
2. Capture the tool traceRecord which tools were called, in what order.
3. Assert on structureRequired tools present; confidence within range; decision matches — never exact text.
4. Skip gracefullyIf the local model server is down, skip rather than fail so other CI jobs aren't blocked.

Pseudocode

FOR EACH case IN golden_set:
  IF model_server down: skip()

  result = run_agent(
    case.input,
    max_iterations = 8,   # no infinite loop
    timeout = 30s)        # fail fast

  # assert on STRUCTURE, not text
  FOR tool IN case.expected_tools:
    assert tool IN result.tools_called
  assert result.decision == case.decision
  assert result.confidence >= case.min_conf

Misconceptions

Use a second model to score what code can't

Some quality dimensions — reasoning coherence, faithfulness to context, evidence use, confidence calibration — can't be checked by a deterministic assertion. The LLM-as-judge pattern feeds the agent's output to a second model that scores each dimension against an anchored scale — five fixed values, each with its own written criteria.

The judge is only as good as its prompt. Anchor every level (a bare "faithfulness 0–1" lets the judge invent what 0.7 means, differently each run), mitigate position bias (randomize order) and length bias (tell it not to reward length), run at temperature 0, and compute the aggregate in code rather than asking the model for it. This is exactly what frameworks like DeepEval and Ragas do internally.

The judge with a scoring rubric

BEFORE: Picture a diving competition where the judge just shouts a gut number after each dive — no rubric, no criteria.

PAIN: Scores swing with mood, the judge favors the diver who went first, and a big flashy splash outscores a clean technical dive. Nobody can trust or reproduce the result.

MAPPING: Give the judge an explicit rubric — entry, execution, difficulty, each scored separately — plus rules like "ignore crowd noise" and "order doesn't matter." An LLM judge works the same way: named criteria and anti-bias instructions turn a gut number into a reproducible score.

Building a good judge

1. Give the judge the evidencePass the question, the agent's answer, and the retrieved context it had access to.
2. Anchor every levelGive 0.00 / 0.25 / 0.50 / 0.75 / 1.00 each its own sentence of criteria, on all four dimensions: reasoning, faithfulness, evidence sufficiency, confidence calibration.
3. Add anti-bias rules"Do not reward longer answers"; randomize order when comparing candidates.
4. Score at temp 0, aggregate in codeThe judge returns four dimensions and nothing else. Your harness snaps each to its nearest anchor and averages them — asking a 7B model for the aggregate hands your threshold back to guesswork.

Pseudocode

prompt = """You are an impartial judge.
QUESTION: {q}   ANSWER: {a}
CONTEXT:  {ctx}
Score 4 dims on these ANCHORS:
 1.00 = ...  0.75 = ...  0.50 = ...
 0.25 = ...  0.00 = ...
Use ONLY those five values.
Do NOT reward longer answers.
No overall - the harness computes it.
Return ONLY JSON."""

reply = judge.chat(prompt, temp=0)
s = parse_json(reply)     # strip ``` fences
s = snap_to_anchors(s)    # 0.8 -> 0.75
s.overall = mean(s.dimensions)
passed = s.overall >= 0.70

Misconceptions

A judge you never checked is not a ruler

Every number in this module is arithmetic on judge output — the 0.70 threshold, the regression baseline, "the average rose from 0.78 to 0.83." All of it comes from a 7B model you have never audited.

A miscalibrated judge does not produce obviously broken output. It produces confident, consistent, wrong numbers, and your whole regression history quietly inherits the error. Two cheap checks — about 14 judge calls, four minutes on CPU, run once per rubric revision — catch it before it costs you six weeks.

The caliper nobody checked

BEFORE: A machine shop buys a digital caliper and starts accepting or rejecting parts with it. It reads to three decimal places, the numbers look authoritative, and for a year nobody questions them.

PAIN: A batch comes back. The caliper had read 0.4mm heavy since the day it arrived — never checked against a gauge block. Every "in tolerance" call for twelve months was wrong in the same direction, and because the readings were consistent, nothing ever looked suspicious.

MAPPING: Your judge is the caliper and your eval suite is the accept/reject line. A silently generous judge, or one quietly rewarding long answers, ships bad parts with green dashboards. Judge validation is the gauge block.

Two checks on the instrument

1. Break a good answer on purposeThree ways, each aimed at one dimension: strip the reasoning, fabricate an EIN that is nowhere in the context, inflate confidence 0.92 → 0.99.
2. Demand a real dropEach break must cost at least 0.25 — one full anchor step. A judge that docks a fabricated answer by 0.05 has technically noticed and practically ignored it.
3. Compare against human labelsScore 10+ cases a human already graded. Report Spearman rho (does it order answers like a human?) separately from mean absolute error (how far off is the level?).
4. Gate, and fix the right thingrho ≥ 0.80 and error ≤ 0.20, non-zero exit in CI. Low rho means rewrite the rubric, not the agent.

Pseudocode

# 1. can it see deliberate damage?
base = judge(good_answer)
FOR EACH break IN [strip_reasoning,
                   fabricate_fact,
                   inflate_confidence]:
  bad = judge(break(good_answer))
  ASSERT base.overall - bad.overall
         >= 0.25   # one anchor step

# 2. does it rank like a human?
h = [c.human FOR c IN labelled]
j = [judge(c).overall FOR c IN labelled]
rho = spearman(h, j)   # ORDER
mae = mean(abs(h - j)) # LEVEL
ASSERT rho >= 0.80 AND mae <= 0.20

Misconceptions

Test the core insight

One question per concept. Tap to reveal the answer.