The 6 concepts of running an agent

Tap any concept to jump to its 5-card cluster.

• 1Production DashboardsFour vital signs on one screen
• 2Alerting: Pages vs TicketsSeverity tiers against alert fatigue
• 3Feedback LoopsReal-time, daily, weekly
• 4A/B Testing & Canary DeploymentsSafety first, then learning
• 5Continuous Improvement CultureA weekly loop, not a crisis response
• 6Agent Versioning & RollbackTagged releases you can revert

Four vital signs for a production agent

A production dashboard turns traces and logs into four questions answered at a glance. Latency: how fast, as p50, p95 and p99. Tokens and cost: per request, and the burn rate per minute. Success and failure: overall and broken down by error type. Drift: is behaviour changing when nobody deployed anything?

Watch the tail, not the average. If 90 of 100 requests take 1.0s and 10 take 8.0s, the average is 1.7s and looks healthy — but the p95 is 8.0s, so one user in twenty waits eight seconds.

The bedside vital-signs monitor

BEFORE: Picture a hospital where a nurse asks each patient “how do you feel?” once an hour and writes “fine” on a clipboard.

PAIN: A patient’s blood pressure could spike and nobody would know until they collapsed. The gap between problem and detection could be fatal.

MAPPING: A dashboard is the bedside monitor: latency, cost, success rate and drift on one screen, updating continuously, with alarms the moment a signal crosses its line.

What each quadrant catches

1. Latency percentilesp50 is the typical user; p99 is your unluckiest 1 in 100.
2. Cost and burn rateA sudden spike often means a prompt regression or a runaway loop.
3. Failures by typeA timeout spike is infrastructure; a guardrail spike is a prompt change.
4. DriftOutput length, tool-call count and tone tracked over rolling windows.

One metric snapshot a minute

EVERY minute:
  snapshot = {
    latency_p50/p95/p99 FROM recent traces
    tokens_per_minute, cost_per_hour
    success_rate
    errors BY type: timeout, guardrail, api
    drift: avg_output_tokens_7d vs baseline
  }
  PLOT snapshot on the four quadrants
  CHECK alert rules AGAINST snapshot

# the dashboard only reads snapshots;
# the traces are the source of truth

Real-time dashboards cut mean time to detection from about 45 minutes to under 4 in Datadog’s study — at 10,000 requests an hour, that’s 7,500 degraded experiences avoided.

Misconceptions

Route every alert by severity

Tiered alerting sorts every alert into a severity and routes it accordingly. P1 / Page: needs a human in minutes — error rate over 50%, agent unresponsive — via PagerDuty or a phone call. P2 / Ticket: next business day — error rate over 10%, p95 over 30 seconds — via Jira or Slack. P3 / Info: trends for a weekly digest, such as output-length drift.

The enemy is alert fatigue: too many noisy alerts, and people start ignoring all of them — including the real ones.

Smoke alarms with one siren

BEFORE: Imagine every smoke alarm in the house had the same deafening siren, whether for burnt toast, a candle near a curtain, or a real kitchen fire.

PAIN: After the third burnt-toast siren, you pull the batteries. Now there’s no protection against a real fire.

MAPPING: Tiered alerting gives three sounds: a quiet chime for toast (P3), a beep for the candle (P2), and a full siren only for fire (P1). People keep listening because the siren means something.

Tuning alerts people trust

1. Start conservativeP1 at 50% errors, P2 at 10% — tighten as the system stabilises.
2. Route by severitySeparate the rules from the routing, so tuning a threshold never touches delivery.
3. Add hysteresisFire only if a condition holds for several consecutive checks, so one spike can’t flap.
4. Prefer relative thresholds“p95 up 30% vs last hour” adapts; a fixed number goes stale.

Rules, routing, hysteresis

RULES = [
  (P1, "error_rate > 0.50", pagerduty),
  (P2, "error_rate > 0.10", jira),
  (P3, "output_drift > 15%", weekly_digest) ]

FOR each rule IN RULES:
  TRY:
    IF rule.condition(metrics):
      rule.streak += 1
    ELSE: rule.streak = 0
    IF rule.streak ≥ 3:  # hysteresis
      ROUTE alert TO rule.channel
  ON error: SKIP rule  # never crash alerting

A few missed low-severity alerts are far safer than a team trained to ignore every page.

Misconceptions

Three speeds of feedback

A feedback loop collects signals about the agent, aggregates them and feeds them into improvement, at three speeds. Real-time: thumbs up or down on each response, attached to its trace. Daily: a script groups the last 24 hours of failures by category for morning triage. Weekly: the most informative failures become new eval cases.

Each speed answers a different question: what broke, what keeps breaking, and did the fix actually stick?

The restaurant that never reads its reviews

BEFORE: A restaurant serves hundreds of meals a day but never reads reviews, never asks diners and never tracks what comes back to the kitchen.

PAIN: Slowly customers stop coming. The chef has no idea the risotto has been underseasoned for months until the reputation is gone.

MAPPING: An agent without feedback is that restaurant. Thumbs-down is the plate sent back; the daily report is reading the reviews; the weekly eval update is changing the recipe so it stays fixed.

The loop

1. CollectA thumbs-down records a score and comment on the trace, with the agent version.
2. AggregateDaily: 187 thumbs-down of 8,412 (2.2%); top category “wrong order referenced”, 23%.
3. ImproveFix the prompt, tool or data behind the top category.
4. Grow evalsEach fixed failure becomes a test case, so it can never quietly return.

From click to test case

ON thumbs_down(trace_id, comment):
  REQUIRE trace_id matches "tr_[a-z0-9]+"
  ATTACH score 0 + comment TO trace

EVERY morning:
  fails = thumbs_down IN last 24h
  GROUP fails BY category
  POST top 5 categories + examples

EVERY week:
  FOR each classified, fixed failure:
    ADD (input, expected) TO eval suite

Validate the trace_id: without it, anyone could post fake scores and poison your dashboards.

Misconceptions

Canary for safety, A/B for learning

A canary deployment sends 1–5% of traffic to the new version and rolls back automatically if its metrics degrade. It asks: does this break anything? An A/B test splits traffic, often 50/50, to measure which version is better. It asks: which one wins?

Do both, in that order: a canary first, then an A/B test. Skip the canary and half your users may meet a broken change; skip the A/B test and you know a change is safe but never whether it’s better.

Don’t replace every engine overnight

BEFORE: An airline develops a more efficient jet engine and fits it to the entire fleet overnight.

PAIN: If the engine has a defect, every plane has it, and there’s nothing to fall back on.

MAPPING: A canary fits the new engine to one plane first. If it performs, it goes on 25% of the fleet, then half, then all. If not, one plane returns to the old engine.

The canary flow

1. Deploy the canaryRoute 1–5% of traffic to the new version.
2. Compare with controlError rate, p95 latency and user feedback, canary vs stable.
3. Auto-rollback gateCanary errors more than 5 points above control roll it back automatically.
4. Promote in stepsAfter a 1–4 hour soak with clean metrics: 25%, then 50%, then 100%.

A sticky traffic router

FUNCTION variant(user_id, canary_pct):
  bucket = SHA256(user_id) MOD 100
  RETURN canary IF bucket < canary_pct ELSE control
  # same user, same variant, every request

EVERY few minutes:
  IF canary.error_rate > control.error_rate + 5%:
    canary_pct = 0  # instant rollback
  ELIF soaked AND all metrics pass:
    canary_pct = NEXT of [5, 25, 50, 100]

Hashing the user ID instead of using random() keeps each user on one variant, so results aren’t muddied and nobody sees the agent’s personality flip mid-conversation.

Misconceptions

Improvement is a process, not an event

An agent’s quality is a moving target: providers update model weights, data changes, user behaviour shifts. Without a repeatable process, quality degrades even if you never change a line of code.

Five pillars keep it improving: auto-score production responses; review the worst 5% weekly and classify each failure; turn every classified failure into an eval case; keep prompts in version control with review; and track improvement velocity, from detection to deployed fix.

Reviewing the game film

BEFORE: A basketball team plays every game but never watches the film afterwards.

PAIN: They keep turning the ball over against the same press and missing the same defensive switch. Teams that review film fix those patterns, and over a season the gap becomes enormous.

MAPPING: The weekly failure review is the game film, the classification is the scouting report, the fix is practice, and the eval run is the scrimmage. Teams that do it weekly compound; teams that only react to crises plateau.

One week’s ticket

1. Pattern“Wrong order referenced” — 43 cases this week.
2. Root causeThe prompt never tells the agent to confirm the order ID first.
3. Fix and testAdd the instruction, plus an eval case: “Where’s my package?” must be answered with a question.
4. Ship and measureCanary at 5%, promoted to 100%; order accuracy 91% → 97%.

The weekly loop

EVERY response: auto_score(correctness,
                            format, safety)
EVERY friday:
  worst = lowest-scoring 5% THIS week
  FOR each case IN worst:
    cause = human CLASSIFY: prompt | tool
                               | data | model
    ADD case TO eval suite
  FIX top causes VIA reviewed PRs
  VERIFY fixes pass the new eval cases
  DEPLOY via canary
TRACK eval score trend + time-to-fix

Teams that run this weekly see eval scores rise 2–5 points a month — often 12–30 points over six months.

Misconceptions

Every change is a tagged, revertible release

Agent versioning treats each combination of prompt text, tool configuration and model parameters as an immutable, tagged release — v2.4.1, just like software. Each tag is stored with its eval results and deployment time.

That makes rollback a single command, not a debugging session. Feature flags add finer control: enable a new tool for 10% of requests, or switch off a misbehaving guardrail instantly, without redeploying.

Save points in a long game

BEFORE: In a long video game you save before every hard boss.

PAIN: Without saves, one bad fight means replaying hours from the start — so players freeze, afraid to try anything new.

MAPPING: Tagged agent versions are save points. Try the new prompt; if it goes wrong, reload the last good tag in seconds and investigate calmly afterwards.

Three interlocking practices

1. Canary gates5% of traffic first; error rate over 3% or eval score under 85% triggers automatic rollback.
2. Feature flagsDeployment and release are decoupled: code is live, behaviour is gated by a flag.
3. Version tagsEvery deploy recorded with its tag, eval scores and timestamp.
4. Revert firstOn a regression, roll back to the last good tag, then investigate.

Deploy, watch, revert

release = TAG "v2.4.1":
  prompt_hash, tool_config, model_params
RECORD release, eval_scores, deployed_at

ROUTE 5% TO v2.4.1
IF error_rate > 3% OR eval_score < 85%:
  ROLLBACK TO last_good_tag  # one command
ELIF soak 30-60 min clean:
  RAMP 25% → 50% → 100%

FLAG new_tool: ON FOR 10% of requests

Teams using this pattern report about 70% fewer production incidents caused by agent updates. Anyone on the team should be able to answer “what changed?” in under a minute.

Misconceptions

Test the core insight

5 questions on the module’s sharpest insights. Tap to reveal the answer.