The 4 concepts of output safety

Tap any concept to jump to its 5-card cluster.

• 1Output ValidationHallucination, toxicity, format checks
• 2Cost ControlsBudget ceilings, loop detection
• 3Human-in-the-LoopApproval, modification, escalation gates
• 4Circuit BreakerStop cascading failures

Check the answer before the user sees it

Input guardrails protect your system from users. Output guardrails protect users from your system. They intercept every agent response before it reaches the user and run three checks.

Hallucination detection verifies each factual claim against the source documents. Toxicity filtering screens for harmful or inappropriate content. Format validation confirms the output matches the expected structure — and it's free, since it's just local parsing, no model call.

The quality inspector on the line

BEFORE: Imagine a factory where products go straight from the machines to the customer, with nobody checking them on the way out the door.

PAIN: Most products are fine, but occasionally a defective one slips through — a car with a faulty brake line, a medicine with the wrong dose. The machine that made it never knew it erred.

MAPPING: Output guardrails are the quality inspector at the END of the line. They don't build the product; they catch the defects manufacturing missed. Your model generates a response, and the guardrail catches the hallucinated fact or malformed output before it ships to the user.

Three checks, cheapest first

1. Format check (free)Parse locally: valid JSON? All required fields present? Values in range? Catches the most common failures with no model call.
2. Hallucination checkA separate model call compares each claim to the source docs and labels it supported, unsupported, or contradicted.
3. Toxicity checkA model-as-judge rates content severity (1–5) and blocks anything over the threshold.
4. Decide the verdictAny contradicted claim or over-threshold toxicity → block. Otherwise pass it to the user.

Pseudocode

FUNCTION validate(response, sources):
  # cheapest check first — no model call
  IF NOT valid_format(response):
    RETURN block("malformed")

  claims = judge_facts(response, sources)
  IF any(c.status == "contradicted"):
    RETURN block("hallucination")

  score = judge_toxicity(response)
  IF score > THRESHOLD:
    RETURN block("toxic")

  RETURN pass(response)

Misconceptions

Cap the spend before a bug drains it

Your agent loop works fine in testing — 3–4 iterations, a couple of cents per request. But an edge case in production can trigger 50 iterations, each chewing through a 100K-token document, turning one request into $15.

Cost controls put a hard ceiling on spend. A middleware layer wraps every model call, tracks cumulative tokens and cost, and before each call checks whether the next one would blow the budget. If so, it short-circuits and returns a graceful fallback instead of continuing to spend.

The teenager's phone bill

BEFORE: Parents hand a teenager a phone on an unlimited plan and never think about it again.

PAIN: Then a $2,000 bill arrives — the teen discovered international calling. The phone worked perfectly; nothing stopped reasonable use from escalating into catastrophic cost.

MAPPING: A budget ceiling is the prepaid card. Each request gets a balance; once it hits zero, the card declines and the agent returns a fallback. Now a runaway loop caps out at a few cents instead of burning the whole monthly budget in an afternoon.

Four levels of control

1. Per-request budgetCap the tokens or dollars for one agent invocation. Exceed it → stop and return a fallback.
2. Per-user budgetDaily or monthly limits per user, so no single power user eats the whole allocation.
3. Loop detectionSpot the same tool being called repeatedly with no progress, and break the cycle.
4. Execution time limitA wall-clock timeout that kills a run stuck in a low-token reasoning loop that never ends.

Which control to add

IF one request can blow up:
  → per-request budget ceiling

IF one user hogs resources:
  → per-user daily / monthly cap

IF same tool fires over and over:
  → loop detection

IF run hangs with few tokens:
  → wall-clock time limit

# middleware sits BETWEEN agent
# and API — logic stays budget-blind

A $0.50 per-request cap allows normal 3–4 iteration flows but stops runaway loops around 8–10 iterations.

Misconceptions

Pause and hand control to a human

Human-in-the-loop (HITL) lets your agent handle routine work autonomously but pause at predefined decision points for a human to review, approve, or modify before proceeding. The human is part of the workflow, not just watching.

There are three gate patterns. Approval gates pause before irreversible actions (send a payment, deploy code). Modification gates present a draft the human can edit. Escalation gates fire when the agent hits its competence boundary and routes to a specialist.

Autopilot and the pilot

BEFORE: Picture a flight where every second demands active human control — no automation to hold altitude, speed, or heading.

PAIN: That's exhausting and error-prone for the routine parts, yet fully removing the human is worse: some moments genuinely need judgment.

MAPPING: Modern autopilot flies the routine hours; the pilot takes over for takeoff, landing, and anything unexpected. HITL works the same way — the agent handles routine requests, but at irreversible or high-stakes points it hands control to a human through a clean, designed handoff.

A gate as a state machine

1. Pause executionAt the gate, the agent stops and stores the pending action in a queue.
2. Notify the humanSend the full context: proposed action, why, and one-click options (approve / deny / modify).
3. Wait for the decisionState sits at waiting_for_human. A timeout can auto-escalate if no one responds.
4. Resume with the answerThe agent continues using the human's decision — execute, use the edited draft, or route onward.

Which gate — and when to escalate

IF action is irreversible
   (payment, deploy, DB write):
  → APPROVAL gate (yes / no)

IF output needs a human voice
   (email, report, letter):
  → MODIFICATION gate (edit draft)

IF policy gap / capability limit
   / explicit request / threshold:
  → ESCALATION gate (route to human)

# escalate on policy gaps, NOT on
# anger. Use programmatic criteria,
# never the model's self-confidence.

Misconceptions

Trip the switch when failures pile up

When failures pile up — API 500s, rate limits, repeated hallucination detections, timeouts — a circuit breaker trips and routes all requests to a safe fallback instead of hammering a failing service.

It has three states. Closed (normal): requests flow, failures counted. Open (tripped): all requests get a fallback, no API calls, a cooldown timer runs. Half-Open (testing): after cooldown, one test request goes through — success closes it, failure re-opens with a longer cooldown.

The breaker in your fuse box

BEFORE: Imagine a house wired with no circuit breaker — just wires running straight from the panel to every outlet.

PAIN: An overload from a faulty appliance or too many devices overheats the wiring and can start a fire. One bad appliance could burn the house down.

MAPPING: The breaker watches the current and TRIPS at dangerous levels, cutting power instantly; you fix the fault and flip it back on. An agent circuit breaker watches failures, trips to a fallback, then tests once before restoring traffic — so one failing dependency can't take everything down.

The three-state cycle

1. Closed — count failuresRequests flow normally. Each failure increments a counter; a success resets it.
2. Trip on thresholdHit the limit (e.g. 3 consecutive) and the circuit opens; a cooldown timer starts.
3. Open — block & fall backEvery request gets an instant fallback. No API calls, no retries, no wasted tokens.
4. Half-Open — test onceAfter cooldown, allow one test request. Success → close. Failure → re-open with a longer cooldown.

Pseudocode

FUNCTION call(request):
  IF state == "open":
    IF cooldown_expired():
      state = "half_open"   # test once
    ELSE:
      RETURN fallback()      # no API call

  result = try_api(request)

  IF result.failed:
    failures += 1
    IF failures >= THRESHOLD:
      state = "open"; start_cooldown()
    RETURN fallback()

  failures = 0; state = "closed"
  RETURN result

Misconceptions

Test the core insight

One question per concept. Tap to reveal the answer.