The 6 concepts of M20

Tap any concept to jump to its 5-card cluster.

• 1Production Monitoring DashboardsLatency, cost, success, drift — on one screen
• 2Alerting: Pages vs TicketsP1 wakes you up, P2 waits till morning, P3 batches
• 3Feedback LoopsThree speeds: real-time, daily, weekly
• 4A/B Testing & Canary DeploymentsShip to 5% before exposing 100%
• 5Continuous Improvement CultureThe weekly loop that keeps the agent ahead of drift
• 6Agent Versioning & RollbackTag every prompt, revert in seconds

Production Monitoring Dashboards

A monitoring dashboard is the vital-signs monitor for your agent. It pulls raw trace data — timestamps, token counts, error codes — and turns it into four real-time charts that answer: How fast? How costly? How reliable? How stable?

The four metric quadrants are latency percentiles, token usage and cost, success/failure rates, and drift detection. Together they paint a complete picture of agent health on a single screen. Without them, you discover problems from angry users — and by then your mean-time-to-detection is already 45 minutes too late.

The hospital vital-signs monitor

BEFORE: A hospital where nurses check on patients by asking "How do you feel?" once an hour and writing a single word — "fine" — on a clipboard. No heart rate, no blood pressure, no oxygen saturation. Just a one-word sample, taken sixty minutes apart.

PAIN: A patient's blood pressure spikes dangerously and nobody notices until they collapse. Subtle deterioration is invisible to hourly snapshots, and by the time someone realises the patient is in trouble, the intervention window has shrunk dramatically. Fifty minutes of unnoticed decline is the difference between a quick fix and a code blue.

MAPPING: A production dashboard is the bedside monitor for your agent. Latency, cost, success rate, and drift stream in real time onto one screen. When any signal crosses a threshold, an alarm fires immediately, not hours later when a user complains. The agent never tells you it's sick — the monitors do.

Four metric quadrants

1. Latency (p50 / p95 / p99)p50 is the typical user experience; p99 is the worst-case 1%. A p50 of 1.2s with a p99 of 15s means one in a hundred users waits 15 seconds — and they're the most likely to churn.
2. Token usage & costTokens consumed per conversation, dollars per request, burn rate per hour. A sudden spike usually means a prompt regression that produces longer outputs or extra tool calls.
3. Success / failure ratePercentage of requests that complete cleanly versus errors (timeout, guardrail block, API failure, malformed output). Always break down by error type so you can tell infrastructure from prompt regression.
4. Drift detectionCompares current output length, tool-call frequency, and sentiment against a rolling baseline. Catches the silent shifts that happen when nobody deployed anything — model weights updated, data drifted, user mix changed.
5. Always group by dimensionTool name, model, user segment, prompt version. Aggregate numbers hide category-level disasters; "errors spike for German names" only shows up in the grouped view.

Mean-time-to-detection drops 12× with a real-time dashboard versus daily check-ins. At 10K req/hour, that's 7,500 saved bad responses every undetected hour.

Pseudocode

What a dashboard config looks like — four panels, all driven from the trace store, refreshed every minute:

# PANEL 1 — latency percentiles
SELECT percentile(latency_ms, 0.50) AS p50,
       percentile(latency_ms, 0.95) AS p95,
       percentile(latency_ms, 0.99) AS p99
FROM traces
WHERE ts > NOW() - 1hour
GROUP BY tool, model       # grouping is the magic

# PANEL 2 — cost burn rate
SUM(input_tokens + output_tokens) AS tokens_per_min,
SUM(cost_usd) AS cost_per_hour

# PANEL 3 — success / failure
COUNT(*) FILTER (WHERE status = "ok") / COUNT(*)
# break down by error_type so you see WHY

# PANEL 4 — drift detection
current = avg(output_tokens) WHERE ts > NOW() - 24h
baseline = avg(output_tokens) WHERE ts BETWEEN NOW()-30d AND NOW()-7d
drift_pct = (current - baseline) / baseline * 100

Every panel is just a SQL aggregation over your trace store. The grouping clauses (BY tool, BY error_type, BY user_segment) are what surface the actionable patterns.

Misconceptions + Takeaway

Alerting: Pages vs Tickets

Tiered alerting classifies every signal into a severity level and routes it to the right channel. P1/Page requires immediate human response (PagerDuty, phone, SMS — wakes you at 3 AM). P2/Ticket needs attention next business day (Jira, Slack). P3/Info goes into the weekly digest.

The single biggest threat to alerting is alert fatigue. When engineers receive too many noisy alerts, they start ignoring all of them — including the critical pages. Strict tiering and conservative initial thresholds keep the signal-to-noise ratio high enough that on-call still trusts the pager.

The smoke alarm severity gradient

BEFORE: Imagine every smoke alarm in your house had the same deafening siren — whether you burned toast, left a candle near a curtain, or had an actual kitchen fire. Every alarm sounds identical, regardless of cause.

PAIN: After the third false alarm from burnt toast, you rip the batteries out. Now you have no protection at all — including against real fires. This is the textbook story of alert fatigue: the noise erodes trust until the engineer treats every page as another piece of toast.

MAPPING: A tiered alerting system is three different alarms. A gentle chime for burnt toast (P3/Info — check the weekly digest). A loud beep for a suspicious candle (P2/Ticket — fix it tomorrow). A full siren for the actual fire (P1/Page — wake someone up NOW). The on-call engineer keeps the batteries in because the loud siren only fires when there's actually fire.

The three tiers in practice

1. P1 / Page — emergenciesConditions that demand response in minutes: error rate >50%, agent fully unresponsive, data breach. Routes to PagerDuty / phone. Acceptable to wake someone at 3 AM.
2. P2 / Ticket — next business dayError rate 10–50%, p95 latency >30s, suspicious cost spike. Routes to Jira ticket or a Slack channel. Investigated tomorrow morning, not tonight.
3. P3 / Info — weekly trendsDrift detected, gradual cost creep, new low-frequency error types. Routes to a weekly email digest or dashboard annotation. Reviewed during regular cadence, never paged.
4. Prefer regression alerts to thresholds"P95 rose 30% in 1 hour vs. last week" beats "P95 > 5s." Threshold alerts page on traffic patterns; regression alerts page on real problems — the kind that correlate with bugs and bad deploys.
5. Start conservative, tighten over timeBegin with loose thresholds (50% for P1, 10% for P2). Once the system stabilises, ratchet down. It's safer to miss a few low-severity alerts than to train the team to ignore everything.

Page or ticket?

Map each alert condition to a tier before it ever fires:

Two questions decide the tier: (1) Will user-facing harm grow if we wait until morning? (2) Is the signal noisy or clean? Yes/clean → P1. No/clean → P2. Either/noisy → P3.

If a P2 fires more than twice a week, demote it to P3 or fix the underlying cause. Pages that fire constantly stop being pages.

Misconceptions + Takeaway

Feedback Loops

A feedback loop collects signals about agent quality, aggregates them, and routes them into processes that improve the agent. The mistake is collecting only one kind of signal — you need three speeds, because each speed answers a different question.

Real-time (thumbs up/down on each response) tells you something went wrong on a specific trace. Daily aggregation reveals patterns ("23% of failures are wrong-order errors"). Weekly eval growth encodes each fix as a permanent test case so the bug never returns. Skip any speed and you fix bugs but never prove they stay fixed — or you spot patterns but can't trace them to individual cases.

The restaurant that never reads its reviews

BEFORE: A restaurant serves hundreds of meals per day but never reads its online reviews, never asks diners how the meal was, and never tracks which dishes get sent back to the kitchen. The chef cooks the same menu year after year, convinced everything is great.

PAIN: Slowly, customers stop coming. The chef has no idea the risotto has been under-seasoned for months or that a competitor across the street is doing it better. By the time revenue drops enough to notice, the reputation is ruined — and the chef still doesn't know which dish was the problem.

MAPPING: An agent with no feedback loop is that restaurant. Users may be hitting subtly wrong answers, confusing edge cases, or unhelpful responses — and you have zero signal to improve. Three feedback speeds (live thumbs-down, daily failure summary, weekly eval growth) are the comment cards, the daily server huddle, and the recipe revisions that keep the kitchen sharp.

Three speeds, one loop

1. Real-time (seconds)Thumbs up/down button on every response. Each click is recorded as a score attached to the trace. Immediate signal on individual responses; the building block for everything else.
2. Daily (hours)Automated script groups the previous 24 hours of negative feedback by failure category, extracts representative examples, posts the summary to Slack. Engineers triage the top categories every morning.
3. Weekly (days)Scheduled job pulls the lowest-scoring 5% of responses, has a human classify them, and adds the most informative cases to the eval dataset (M18). Your test suite grows organically from real production failures.
4. Capture implicit signals tooUsers who silently re-ask, abandon mid-flow, escalate to support, or copy-paste-into-Google rarely click the thumbs-down button. Build the implicit pipeline; don't wait for the explicit click.
5. Close the loopEvery fix becomes an eval case. Every eval case validates the next prompt change. Every prompt change generates fresh feedback. Skip the eval-case step and you fix bugs without proving they stay fixed.

Teams running all three speeds report 40–60% fewer repeated failures within the first month.

Pseudocode

The end-to-end collector that turns a thumbs-down into a future regression test:

# REAL-TIME — capture each click
ON user_feedback(trace_id, score, comment):
  STORE feedback {trace_id, score, comment, ts}
  IF score == thumbs_down:
    QUEUE trace_id FOR daily_review

# DAILY — aggregate & classify
EVERY 24h:
  failures = LOAD daily_review_queue
  groups = CLASSIFY failures BY failure_category
  POST TO slack:
    "Top 5 failure categories last 24h"
    FOR EACH group IN top_5(groups):
      print(group.name, group.count, group.example_trace)

# WEEKLY — grow the eval dataset
EVERY 7d:
  worst = SELECT bottom 5% scored traces last week
  reviewed = HUMAN_LABEL(worst)
  FOR EACH case IN reviewed:
    ADD {input, expected_output} TO eval_dataset
  RUN_EVAL(eval_dataset, current_prompt)

Notice how each speed feeds the next: the thumbs-down queues a daily case, the daily case becomes a weekly eval, the eval gates the next deploy.

Misconceptions + Takeaway

A/B Testing & Canary Deployments

A canary deployment is a safety strategy: send 1–5% of traffic to the new version, monitor for errors, auto-rollback if metrics degrade. Goal: does this change break anything? An A/B test is an experiment: split traffic 50/50 and measure which version is better on a specific metric. Goal: which version is better?

The two patterns are complementary, not interchangeable. In practice you canary first (1–5% to check for regressions) and then expand to an A/B test (50/50 to measure quality). Skip the canary and you risk exposing half your users to a broken change. Skip the A/B test and you know your changes are safe but never know whether they're actually better.

Don't replace all engines mid-flight

BEFORE: An airline develops a new, more efficient jet engine. Instead of testing it on one aircraft first, they ground the entire fleet overnight and replace every engine on every plane simultaneously.

PAIN: If the new engine has any defect — a manufacturing flaw, a software bug, an interaction with the existing airframe — every single plane is affected. There are no working aircraft to fall back on. Passengers are stranded, and the airline faces a catastrophic safety incident with zero buffer.

MAPPING: Deploying a new prompt or model version to 100% of traffic is the same gamble. A canary sends just 5% to the new version — if metrics regress, auto-rollback hits in seconds with minimal user impact. If the canary survives a soak period, you ramp to 25%, 50%, and finally 100%. The first plane carries 5% of the passengers, not all of them.

The canary → A/B pipeline

1. Deploy canary at 5%Route a hash-based slice of traffic to the new version. Hash on user_id so the same user keeps hitting the same version — consistency matters for conversational agents.
2. Soak for 30–60 minutesCompare canary vs. control on golden signals: error rate, p95 latency, user feedback score. The soak period gives flaky metrics time to settle.
3. Auto-rollback gateIf canary error rate exceeds control by >5pp, or p95 doubles, or feedback score drops below floor — route 100% back to stable in seconds. No human in the loop.
4. Promote in stages5% → 25% → 50% → 100%, with a soak at each step. Each stage is another chance for a slow regression to show up before the blast radius grows.
5. Then run the A/B testOnce the canary proves the change is safe, run a 50/50 A/B test to measure whether it's actually better. Plan for 3–5× more samples than a typical web A/B test — LLM non-determinism inflates variance.

Pseudocode

The canary state machine that protects every prompt deploy:

# traffic split: hash(user_id) decides which version
DEFINE route(user_id):
  IF hash(user_id) % 100 < canary_pct:
    RETURN canary_version
  RETURN control_version

# promotion ladder with auto-rollback
deploy(canary_v="v2.4", canary_pct=5)

FOR stage IN [5, 25, 50, 100]:
  set canary_pct = stage
  WAIT soak_period (30–60min)

  c = metrics(canary_version, last=soak_period)
  k = metrics(control_version, last=soak_period)

  IF c.error_rate - k.error_rate > 0.05
     OR c.p95_latency > 2 * k.p95_latency
     OR c.feedback_score < 0.60:
    ROLLBACK TO control_version    # instant
    PAGE on_call("canary regressed at " + stage + "%")
    EXIT

PROMOTE canary_version TO stable

The state machine has just two outcomes per stage: pass and ramp, or regress and roll back. Never "pass and stay" — partial rollouts left in place become silent technical debt.

Misconceptions + Takeaway

Continuous Improvement Culture

Continuous improvement is the practice of systematically making your agent better over time through structured processes, not ad-hoc heroics. Unlike traditional software where you ship a feature and move on, an AI agent's quality is a moving target — user expectations shift, data changes, and model providers update weights silently.

Without a repeatable improvement process, your agent's quality will degrade even if you never change a line of code. The five pillars are auto-scoring production traffic, weekly failure reviews, growing eval datasets from real failures, version-controlling prompts, and tracking improvement velocity. Run them on a calendar, not on crisis.

Elite teams review game film

BEFORE: A professional basketball team plays games but never watches game film afterward. They never analyse which plays worked, which defensive rotations failed, or which opponent strategies surprised them. They just show up for the next game and hope for the best.

PAIN: They keep making the same mistakes — turning the ball over against full-court presses, missing the same defensive switch. Teams that do review film identify these patterns and drill fixes. Over a season, the gap between film-reviewers and non-reviewers becomes enormous.

MAPPING: Continuous improvement for an agent follows the same playbook: systematically review failures (game film), identify patterns (scouting report), implement fixes (practice), and verify them with evals (scrimmage). Teams that do this weekly compound improvements; teams that only react to crises plateau.

Five pillars on a weekly cadence

1. Auto-score production trafficRun M18-style evals on a sample of every day's real responses, not just on test data. Production becomes a continuous evaluation environment.
2. Weekly failure reviewEvery Friday, pull the lowest-scoring 5% of responses. A human classifies each: prompt problem, tool problem, data problem, or model limitation. Classification drives targeted fixes.
3. Growing eval datasetsEvery classified failure becomes a new test case. Over time, your eval dataset becomes a comprehensive map of every failure mode the agent has ever encountered.
4. Version-controlled promptsStore prompts in Git alongside the code. Every prompt change gets a PR, a review, and an eval run before merging. Prevents prompt drift — ad-hoc edits accumulating without documentation.
5. Track improvement velocityTime from failure detection to fix deployed. Weekly eval-score trend. The goal isn't perfection; it's consistent, measurable improvement — 2–5 points per month is a healthy pace.

Where does this failure go?

Classifying failures correctly is what makes the weekly review actionable. Use this matrix for each low-scoring trace:

Every classification creates exactly one eval case and exactly one fix. If a failure doesn't fit a category, it usually means the symptom is masking two problems — split it.

Misconceptions + Takeaway

Agent Versioning & Rollback

An agent's behavior is determined by the combined state of its system prompt, tool schemas, model parameters, and feature flags. Agent versioning treats every combination as an immutable, tagged release — just like a software version — so you have a concrete target to revert to when something goes wrong.

Three interlocking practices make rollback fast and confident: canary gates limit the blast radius of new versions, feature flags let you toggle individual capabilities without redeploying, and version tags tied to eval results mean any team member can answer "what changed?" in under a minute. Rollback itself is a single command, not a manual process.

Git tags for behavior, not just code

BEFORE: A team that "deploys" agent changes by editing the system prompt directly in a config file in the running container. There's no version number, no commit log, no record of what was different an hour ago.

PAIN: When the agent suddenly starts hallucinating tracking numbers at 2 PM Tuesday, nobody can answer "what changed?" The on-call has to grep terminal history, ask everyone on Slack, and guess. The rollback is a manual prompt edit performed under stress — the most error-prone moment to make a careful text change.

MAPPING: Tagging every prompt+tool+model combination as v2.4.1 turns an opaque mess into a Git history. Eval scores attach to each tag. When the dashboard turns red, you don't debug under pressure — you revert to v2.3.7 with one command and investigate later when the on-call isn't on fire. Same idea as git revert, but for runtime behavior.

Tag, gate, flag, revert

1. Tag every releaseA version is the full bundle: system prompt + tool schemas + model + temperature + flag state. Each gets a semver tag (v2.4.1) recorded in your config store.
2. Attach eval results to the tagBefore promotion, run your eval suite and store {tag, eval_score, deploy_ts} in observability. Now "what was the eval score on Tuesday at 2 PM?" is a one-line lookup.
3. Canary gateNew tag goes to 5% of traffic. Auto-rollback if metrics regress vs. control during the soak period (see Cluster 4). The canary gate is the entry point to the rollback machinery.
4. Feature flags for fine-grained controlToggle individual capabilities without redeploying: enable a new tool for 10% of requests, disable a problematic guardrail instantly, expose a new prompt clause to internal testers only. Decouples deploy from release.
5. One-command rollbackagent rollback v2.3.7. Routing flips in seconds. No manual prompt edit, no config file scramble — the rollback is as boring and reliable as the deploy.

Teams adopting tagged versions + canary gates report 70% fewer production incidents caused by agent updates.

Pseudocode

The lifecycle every agent version moves through — from draft to stable to retired:

# a version is the FULL bundle
DEFINE version v2.4.1 = {
  prompt:      "prompts/system_v2.4.1.md",
  tools:       ["order_lookup_v3", "refund_v2"],
  model:       "claude-sonnet-4-6",
  temperature: 0.2,
  flags:       {empathy_clause: true}
}

# state machine: DRAFT → CANARY → STABLE → RETIRED
STATE draft:
  RUN_EVAL(v2.4.1) → record(eval_score)
  IF eval_score < previous_stable.score: BLOCK
  ELSE: promote_to(canary)

STATE canary:           # 5% → 25% → 50% → 100%
  monitor(soak_period)
  IF regression_detected:
    rollback_to(previous_stable)   # one command
    PAGE on_call
  ELSE: ramp_to(next_stage) OR promote_to(stable)

STATE stable:
  ALL_TRAFFIC → v2.4.1
  previous_stable → retired (kept for fast rollback)

Notice the previous stable version is retired, not deleted. Keeping the last 2–3 versions hot lets you roll back in seconds even if the build pipeline is down.

Misconceptions + Takeaway

Tap a card to reveal the answer