The 5 concepts of monitoring

Tap any concept to jump to its 5-card cluster.

• 1Monitoring vs EvaluationOffline gate vs online watch
• 2The Four Golden SignalsLatency, errors, tokens, quality
• 3Production SamplingScore 5–10% representatively
• 4Model Drift DetectionCatch silent regressions
• 5Closing the Feedback LoopFailures become eval cases

Two different questions

Evaluation is offline: run a fixed test set against your agent before a change ships, measure quality, and gate the release on a pass threshold. It answers “is this version good enough to ship?”

Monitoring is online: the agent is serving real traffic and you continuously collect metrics, sample outputs, and alert on anomalies. It answers “is what I shipped still working?” Neither replaces the other — a perfect test set can't anticipate every real input.

The health inspector

BEFORE: A health inspector tests a restaurant once, before it opens. The kitchen passes with flying colors — clean surfaces, right temperatures, trained staff — and the inspector signs the certificate and leaves.

PAIN: Six months later the fridge drifts two degrees, a vendor changes a recipe, a new hire cuts a corner. Each change is tiny; none would have failed opening day. Then fifty people get food poisoning on a Tuesday.

MAPPING: Evals are the opening inspection. Monitoring is the health department doing random spot checks every week while the restaurant is serving. You need both: evals to confirm you're ready to open, monitoring to confirm you stay safe under real load.

Where each one lives

1. Eval runs before deployA curated golden set runs against the new version; 100% coverage, minutes to complete, gates the merge.
2. Ship the passing versionPassing evals means “ready to open” — not “guaranteed correct forever.”
3. Monitor live trafficContinuously collect signals on real requests; sample 5–10% for quality scoring.
4. Alert in secondsAnomalies (drift, new failure modes, spikes) page you in real time, not after a customer complains.

Eval or monitor?

IF question is "ready to ship?"
   (before a deploy, fixed test set):
  → run an EVAL — offline, 100% coverage

IF question is "still working?"
   (after deploy, real traffic):
  → MONITOR — online, sampled, alerting

# they answer different questions —
# you always need BOTH, not one.

Monitoring without evals gives signals but no root-cause anchor; evals without monitoring miss everything that changes after release.

Misconceptions

Four signals, four failure classes

Google's SRE book defines four golden signals for any system: latency, traffic, errors, saturation. For LLM agents you swap two: traffic→token spend and saturation→quality score, because those are where agents fail in ways ordinary services don't.

So the minimum useful set is latency, error rate, token spend, and quality score. Each maps to a distinct failure class, and each fails first for a different kind of problem — so watching all four catches trouble no single metric would.

The cockpit instruments

BEFORE: You deployed and it works. Congratulations — you're now flying a plane, but you've been staring out the windshield instead of the instrument panel.

PAIN: In clear weather that's fine. But drift, a slow tool, or quietly worse answers are invisible from the window. Without instruments you only learn something's wrong when passengers start screaming.

MAPPING: The four signals are your cockpit gauges: airspeed (latency), warning lights (errors), fuel burn (token spend), and altitude (quality). One glance at four dials tells you if you're flying level — long before anyone in the cabin notices.

What each signal catches

1. Latency (p50/p95/p99)Track percentiles, not the average. p95 = your unluckiest 1-in-20 user. Confused model → more tool calls → p99 spikes first.
2. Error rateSplit by type: tool failure, loop timeout, parse error. A model update that changes output format makes parse errors spike first.
3. Token spendTokens per query/session. On local Mistral the cost is GPU time, but runaway growth backs up the queue either way.
4. Quality scoreLLM-as-judge on sampled traffic. The only signal that catches silent degradation — latency fine, answers quietly worse.

Pseudocode

# declare metrics once, at startup
requests = COUNTER(by=status)
errors   = COUNTER(by=error_type)
latency  = HISTOGRAM(buckets)  # p50/95/99
tokens   = HISTOGRAM(buckets)
quality  = GAUGE()

WRAP every agent run:
  start = now()
  TRY run agent
     requests(status="ok").inc()
  CATCH err
     errors(type=err.kind).inc()
  FINALLY
     latency.observe(now() - start)
     tokens.observe(total_tokens)

# scraper pulls /metrics every 15s;
# p95 comes from the histogram buckets.

Misconceptions

Score a smart slice, not everything

You can't run an LLM-as-judge on every request — at 1,000 queries/day that doubles your inference load. So you score only 5–10% of traffic.

The catch: the slice must be representative, not just the first N requests of the day. Reservoir sampling keeps a fixed-size pool where every request — peak or off-peak — has an equal chance of being picked, so the sample mirrors the whole day. Errors are always kept, because they're the most informative runs.

Quality control on a factory line

BEFORE: A factory inspects every single item that comes off the line for defects — thorough, but it needs as many inspectors as it has output.

PAIN: Doubling the workforce just to check is unaffordable. So a lazy team inspects only the first hour's items — and misses the night-shift batch where the real defects appear.

MAPPING: Reservoir sampling is the smart QC clerk with a fixed tray of, say, 50 slots. As items stream by all day, each has an equal shot at landing in the tray — morning or night shift. Inspect just the tray and you've fairly represented the entire run at a fraction of the cost.

The sampling pipeline

1. Every run streams inEach completed agent run is offered to the sampler along with an is-error flag.
2. Reservoir decidesFill the reservoir first; after that, each new run has a fair chance of evicting a random slot. Errors are always kept.
3. Judge the sampleThe kept runs go to the LLM judge (also Mistral-7B), scored 0–1 on relevance, faithfulness, and completion.
4. Store & alertWrite the score to the quality gauge and flush samples to disk; alert if the average drops below 0.7.

Pseudocode

reservoir = []       # fixed size, e.g. 50
seen = 0

FUNCTION record(run, is_error):
  seen += 1
  IF is_error:
     keep run  # errors are gold, always keep
     RETURN
  IF reservoir not full:
     reservoir.append(run)
  ELSE:
     j = random(0 .. seen-1)     # Vitter's Algo R
     IF j < size: reservoir[j] = run

# every run — peak or 3am — has an
# equal chance of being in the sample.

Misconceptions

Catch the slow, silent slide

Model drift is when outputs change in quality or distribution versus a known baseline — even when the weights are nominally the same. A careless ollama pull mistral can swap weights silently; user inputs evolve; tool schemas and context quality shift.

Drift is insidious because each change is imperceptible — only the cumulative effect is measurable. Three reliable local signals: rolling quality score, token count (confused models ramble longer), and tool-call frequency (a struggling model stops using tools).

The frog in warming water

BEFORE: You know exactly how your agent behaved at launch — a clear baseline of scores, token counts, and tool usage.

PAIN: Nothing ever breaks loudly. The water warms one degree at a time: a slightly newer quantization, gradually shifting queries, subtly worse retrieval. No single day looks wrong, so you never jump — until you're two months in and quality is quietly cooked.

MAPPING: A drift detector is the thermometer that ignores the absolute temperature and watches the rate of change. It compares today against a rolling baseline and alarms when the shift is bigger than normal wobble — so you leap out before you boil.

Z-score against a rolling baseline

1. Keep a rolling windowStore recent values (e.g. 7 days) for quality, tokens, and tool-call frequency.
2. Split baseline vs currentThe older half is the “baseline”; the newest ~20% is “current.”
3. Compute a z-scoreHow many standard deviations has the current mean moved from the baseline mean?
4. Threshold & fire|z| ≥ 2 → warning; |z| ≥ 3 → critical. This tolerates daily wobble but catches real shifts.

Pseudocode

FUNCTION check_drift(window):
  IF window too small: RETURN none

  baseline = older_half(window)
  current  = newest_20pct(window)

  mean_b = mean(baseline)
  std_b  = stdev(baseline)
  IF std_b ~ 0: RETURN none  # flat

  z = (mean(current) - mean_b) / std_b

  IF abs(z) >= 3: RETURN "critical"
  IF abs(z) >= 2: RETURN "warning"
  RETURN none

# run per metric: quality, tokens,
# tool-call frequency. Persist window
# to disk so it survives restarts.

Misconceptions

Failures become tomorrow's tests

The feedback loop is the operational bridge between monitoring and evaluation. Every failure your monitoring catches is a candidate new eval case; every quality drop is a signal to augment your golden set.

Without the loop your eval set silently goes stale — a perfect test for yesterday's failure modes. Close it and the cycle turns: deploy → monitor → detect drift → collect failures → fix & re-eval → deploy again. A thumbs-down on a real query is worth more than ten synthetic tests.

The airline incident review

BEFORE: Aviation is astonishingly safe. When something goes wrong, the crew doesn't just patch it and move on — every incident is logged and studied.

PAIN: A team that fixes each production glitch in isolation learns nothing durable. The same class of failure resurfaces months later because the checklist never changed.

MAPPING: Closing the loop is the incident review board: each real failure becomes a permanent item on the pre-flight checklist (your eval set). The next release is tested against every past failure — so the system gets monotonically safer instead of repeating its mistakes.

From thumbs-down to eval case

1. Collect feedbackCapture thumbs-up/down from the UI plus full run context: query, response, tool calls, tokens.
2. Log to JSONLAppend each record to a file compatible with the eval harness — the cheapest labeled data you'll ever get.
3. Debounce the triggerAccumulate N new failures (or wait an hour) before acting — never re-eval on every single thumbs-down.
4. Ingest & re-evalConvert failures into eval cases, run the harness, fix, and redeploy. The loop closes.

When to trigger a re-eval

ON each thumbs-down:
  log full run to feedback.jsonl
  new_failures += 1

IF new_failures >= threshold (e.g. 20)
   AND last_trigger was > 1 hour ago:
  → ingest failures into eval set
  → run the eval harness
  → reset counter + timer

# debounce is the whole game:
# trigger on EVERY down = thousands
# of eval jobs/day. Batch instead.

Rate-limit the loop: accumulate a batch or cap at once per hour, or you'll drown your CI in eval runs.

Misconceptions

Test the core insight

One question per concept. Tap to reveal the answer.