The 7 concepts of agent observability

Tap any concept to jump to its 5-card cluster.

• 1Why Observability MattersAgents are non-deterministic; "it broke" tells you nothing
• 2Agent TracesEnd-to-end record of one request, identified by trace_id
• 3Spans: Nesting & TimingThe sub-units inside a trace — LLM call, tool call, guardrail
• 4Structured LoggingJSON logs with typed fields, queryable from day one
• 5Observability ToolsLangfuse, OpenTelemetry, Datadog, Honeycomb — pick wisely
• 6Compliance & Audit LoggingPII redaction, immutable storage, retention windows
• 7Prompt Management & VersioningTreat prompts as code — Git, A/B tests, regression suites

"It broke" tells you nothing

A single agent run can issue 5–50 LLM calls, 10+ tool invocations, retries, fallbacks, and conditional branches. When something goes wrong in production, "it broke" gives you no path to a fix. You need to reconstruct the exact sequence of decisions Claude made — what it saw, what it called, what came back, and how long each step took.

Agents are non-deterministic: the same input can produce different tool calls and reasoning paths each run. Traditional "set a breakpoint and reproduce" debugging fails. Observability sidesteps this entirely — every execution is captured as it happens, leaving a permanent, queryable record you can analyse after the fact.

Driving at night with no dashboard

BEFORE: Imagine driving a car at night with no headlights, no speedometer, no fuel gauge, no GPS. You press the accelerator and hope you're going the right speed in the right direction.

PAIN: When something goes wrong — you drift off course, the engine starts knocking, the tank empties — you have no way to know what happened until you crash. Every incident becomes a guessing game; every fix is "let me try something and see."

MAPPING: Observability is your headlights, dashboard, and GPS combined. Traces are the road your agent took (every step in sequence). Spans are timing for each turn and acceleration. Structured logs are the gauges — real-time readings of speed, fuel, engine temp. With these instruments you see exactly what happened, when, and why.

Three reasons agents need MORE than a normal app

1. Non-determinismThe same prompt can yield different tool calls and reasoning paths on each run. Unlike a REST API that returns the same JSON for the same input, Claude might call search_orders on one run and check_inventory on another. Traces show which path was taken for each specific request.
2. Multi-step chainsAgents make 5–20 decisions per request. A bug in step 7 may only manifest as a wrong answer in step 12, with five correct steps in between hiding the root cause. Without span trees, the symptom and cause are pages apart.
3. Cost visibilityEach LLM call costs $0.003–$0.015. A runaway loop (50 calls instead of 3) can burn the daily budget before anyone notices. Per-call token counts in traces are how you spot the spike in real time.
4. The payoffTeams without observability spend 4–8 hours debugging a single agent failure. Teams with proper tracing resolve the same issue in 10–15 minutes — they pull up the trace_id and see the exact sequence of decisions that led to the problem.

When do I need this?

# Question: should I bother with tracing now?

IF agent is a single-shot prototype on your laptop:
  SKIP — print() is fine for now
  # but add tracing BEFORE first deploy

ELIF agent will see real users (any volume):
  ADD tracing on day one
  # retrofitting later costs 10x more

ELIF agent loops (multi-step, multi-tool):
  ADD spans + structured logs immediately
  # you cannot debug a loop from print logs

ELIF agent handles regulated data (PHI, PII, $$):
  ADD tracing + audit logs + redaction
  # compliance is non-negotiable — see Cluster 6

ELIF P95 latency or cost matter:
  ADD per-span timing + token counts
  # can't optimise what you can't measure

ELSE:
  ADD tracing anyway. The cost is <1ms / call.

There is no realistic case where shipping an agent without tracing pays off. Async export is essentially free; the lessons it unlocks are not.

Misconceptions

One trace = one request, end to end

A trace is the complete record of a single agent execution — from the moment a user request arrives to the moment a response is sent. It has a globally unique trace_id, a start and end timestamp, the input, the output, and a tree of spans capturing every sub-step inside.

The trace_id is the thread that ties 47 events together as ONE request. Without it, log lines from concurrent users interleave randomly and causality is lost. With it, you can pull up any historical run by ID and see the entire story — including which LLM call was slow, which tool returned the wrong data, and what the final answer was.

The flight black box

BEFORE: Early aircraft had no recording. Investigators relied on witness statements and twisted wreckage to guess what happened in the cockpit during the final two minutes.

PAIN: Without instrumentation, the same crash happens again because no one can prove the root cause. Every incident is a guessing game; lessons are lost; aviation stalls.

MAPPING: The modern flight data recorder logs every sensor reading, every control input, every system state, with synchronised timestamps. After an incident you replay the timeline and see exactly which valve opened in which order. A trace is your agent's flight recorder. Every span is one sensor reading. The trace_id is the flight number. When a user complains "the agent gave me garbage at 3:14pm yesterday," you load the trace and watch what happened.

What every trace must capture

1. IdentityA globally unique trace_id (UUID), assigned at the entry point and propagated through every nested call.
2. Timingstart_time and end_time on the trace and on every span. Total duration_ms falls out for free.
3. Input & outputThe user's message in, the final response out (hashed or summarised if it contains PII).
4. Span treeEvery LLM call, tool call, RAG search, and guardrail becomes a child span with parent_span_id pointing up the tree.
5. AggregatesTotal cost, total tokens, error status, user ID hash. The numbers you'll dashboard later.

Traces follow the OpenTelemetry standard — the same shape works in Python, TypeScript, Go, and any backend (Langfuse, Honeycomb, Datadog, Jaeger).

Pseudocode

An instrumented agent run — every step nested under the root, attributes captured, async export at the end:

FUNCTION handle_request(user_input):
  trace_id = NEW_UUID()
  WITH span("agent.run", trace_id, parent=NULL) AS root:
    root.attrs = {user_id, input_len: len(user_input)}

    WHILE NOT done:
      WITH span("llm.call", trace_id, parent=root) AS s:
        response = claude.messages.create(...)
        s.attrs = {model, tokens_in, tokens_out, cost}

      IF response.has_tool_call:
        WITH span("tool." + name, trace_id, parent=root) AS t:
          result = run_tool(name, args)
          t.attrs = {args, result_size, latency_ms}
      ELSE:
        done = TRUE

    root.attrs["total_cost"] = SUM(s.cost for s)

  EXPORT trace TO backend  # async, non-blocking

Every span carries its trace_id and parent; attributes are typed key/value pairs (not free-form strings); export is the LAST thing — it must never block the user response.

Misconceptions

Spans are the building blocks inside a trace

A span is one unit of work inside a trace — a single LLM call, a single tool invocation, a single guardrail check. Each span has a unique span_id, a parent_span_id (the span that triggered it), a name, start/end timestamps, a status (ok / error), and typed attributes like llm.model, llm.input_tokens, tool.name.

The parent-child relationships create a tree that mirrors the actual execution flow. Visualised as a flame graph, the tree shows you instantly which step ate the latency, which step failed, and how everything depends on everything else.

Recipe steps inside a recipe

BEFORE: Picture cooking lasagna. "Make lasagna" is the top-level task. Inside it: "make sauce" (chop onions, brown meat, simmer), "make pasta layers" (boil noodles, drain), "assemble and bake."

PAIN: If your lasagna tastes wrong, you need to know which sub-step failed. Burnt onions? Overcooked noodles? Oven too hot? Without timing each step, you taste the final dish and guess backwards.

MAPPING: A span is one recipe step. The root span ("handle user request") contains child spans ("call LLM," "execute tool," "validate output"). Each child can have its own children ("parse JSON" inside "execute tool"). The nesting shows you exactly where time is spent and where errors originate — a flame graph for your kitchen.

The five span types every agent emits

1. Root span (agent)The entire request lifecycle. Captures total latency, overall status, final output summary. Every other span is a descendant.
2. LLM spanEach call to Claude. Attributes: model, input_tokens, output_tokens, latency_ms, cost_usd, stop_reason. Usually the most expensive span by both time and money.
3. Tool spanEach tool invocation — DB query, API call, file read. Attributes: tool.name, args summary, result size, success/failure, latency. Where bottlenecks usually hide.
4. Retrieval spanRAG searches. Attributes: query, k, top scores, doc IDs returned. Critical for "the agent gave a wrong answer" debugging — usually the retriever picked the wrong docs.
5. Guardrail spanInput/output validation checks. Attributes: check name, pass/fail, reason. Fast (<10ms) but the pass/fail is essential for compliance audits.

Width = duration. Color = status. Tree depth = call hierarchy. That's the entire visual language.

Pseudocode

A span as a context manager — auto-times itself, auto-records errors, auto-links to parent:

CLASS Span:
  FUNCTION __enter__():
    self.span_id = NEW_UUID()
    self.start = now()
    self.parent_span_id = current_span_in_context()
    PUSH self ONTO context_stack
    RETURN self

  FUNCTION __exit__(error):
    self.end = now()
    self.duration_ms = self.end - self.start
    self.status = "error" IF error ELSE "ok"
    POP self FROM context_stack
    queue_for_export(self)  # async

# Usage — nesting is implicit via the context stack
WITH span("agent.run") AS root:
  WITH span("llm.call") AS s:    # parent = root
    s.set("llm.model", "sonnet-4-6")
    s.set("llm.input_tokens", 1840)
    response = claude.messages.create(...)

Context-managed spans are why OpenTelemetry "just works" — the SDK tracks the active span per thread/coroutine and links new spans to it automatically.

Misconceptions

JSON, not strings

A structured log is a JSON object — not a sentence. Instead of "LLM call took 847ms", you emit {"event":"llm_call","trace_id":"tr_a1b2","latency_ms":847,"model":"sonnet-4-6","tokens_in":1840}. Every field is a typed column you can query: "show me P95 latency_ms for model = sonnet-4-6 in the last hour" becomes a one-line SQL question instead of a weekend grep session.

Structured logs are the dashboard gauges that pair with traces. Traces show you the shape of one run; logs let you aggregate across millions of runs to spot trends, alert on regressions, and feed dashboards. The two are siblings, not substitutes.

Filing cabinet vs junk drawer

BEFORE: Picture a junk drawer — receipts, batteries, takeout menus, screwdrivers, all jumbled together. Need a 2019 receipt for taxes? You're tipping the drawer onto the floor.

PAIN: A grep across print() output is the junk-drawer experience at 3am. Lines from concurrent users interleave, formats vary by who wrote the line, fields are buried inside English sentences. There's no schema, so nothing is queryable.

MAPPING: A filing cabinet has one folder per category, each folder has labelled tabs, each tab has a date. Structured logs are the cabinet: every entry is JSON, every event has a name, every field has a type. SELECT * WHERE event='llm_call' AND latency_ms > 2000 AND date > today()-1 is the moral equivalent of "open the LLM folder, find the slow tab, last 24 hours."

The four rules of structured logs

1. Emit JSON, not textOne log line = one JSON object. Every line parses with JSON.parse(). No regex needed downstream.
2. Always include the trace_idThe same trace_id on every log line is what stitches log entries to a trace and to each other across processes and machines.
3. Standardise field namesPick one schema for the agent: event, trace_id, step, model, tokens_in, tokens_out, latency_ms, cost_usd, stop_reason. Same names everywhere, every time.
4. Log decisions, not narrationCapture the moments that change behaviour: tool chosen, retry triggered, guardrail tripped, fallback used. Don't log "entering function foo()" — you have spans for that.

Use structlog (Python) or pino (Node) — both emit JSON by default and pair cleanly with any backend.

Pseudocode

One emitter, JSON output, every entry carries its trace_id:

# Configure once at startup
logger = JSONLogger(
  fields=["timestamp", "level", "event", "trace_id"],
  output=stderr  # keeps stdout for user response
)

# Inside the agent loop
logger.info(
  event="llm_call",
  trace_id=trace_id,
  step=step,
  model="claude-sonnet-4-6",
  input_tokens=resp.usage.input_tokens,
  output_tokens=resp.usage.output_tokens,
  latency_ms=elapsed_ms,
  cost_usd=cost,
  stop_reason=resp.stop_reason,
)

# Outputs one line of JSON, e.g.:
# {"timestamp":"2025-03-15T14:32:02.694Z","level":"info",
#  "event":"llm_call","trace_id":"tr_a1b2c3d4","step":1,
#  "model":"claude-sonnet-4-6","input_tokens":1840,
#  "output_tokens":89,"latency_ms":847,
#  "cost_usd":0.0072,"stop_reason":"tool_use"}

Pipe stderr to a file or to your aggregator (Loki, ELK, ClickHouse). One config, every line queryable.

Misconceptions

Two big choices: LLM-native or platform-standard

The agent observability ecosystem has matured into two families. LLM-native tools (Langfuse, LangSmith, Arize Phoenix) understand what an LLM call is, calculate cost per model out of the box, and ship prompt playgrounds and eval datasets. Platform-standard stacks (OpenTelemetry + Honeycomb / Datadog / Grafana Tempo) don't know about tokens but plug into the existing APM your platform team already runs.

Most production teams run both: OTel feeds enterprise APM dashboards (SLOs, on-call alerts, full request waterfall), and Langfuse owns the LLM-native workflow (prompt diffing, cost rollups, eval datasets). They are complements, not competitors.

Specialist clinic vs general hospital

BEFORE: If your knee hurts you can go to a general hospital ER or to a sports-medicine specialist. Both are doctors; both can help. They optimise for different things.

PAIN: Pick only the specialist and you're stranded when you also need a flu shot or a cardiogram. Pick only the general hospital and you wait six hours for someone who has seen 200 knees this year.

MAPPING: Langfuse is the sports-medicine clinic for your agent — deep expertise on the LLM-specific knee problem (prompt diff, cost per model, eval dataset). OpenTelemetry + Honeycomb is the general hospital — it sees every system from the load balancer to the database to the agent in one waterfall. Production teams keep both numbers in their phone.

How to choose

1. Start with what your platform team already runsIf Datadog, Honeycomb, or Grafana Tempo is already wired up, instrument with OpenTelemetry and ship LLM spans into the same pipeline. Zero new vendors, zero new dashboards.
2. Add an LLM-native layer when prompt iteration mattersThe moment you start A/B testing prompts or running evals, Langfuse (open source, self-host) or LangSmith (cloud, LangChain-native) earn their keep with prompt playgrounds, dataset diffing, and per-model cost.
3. Match data residency to the use caseRegulated workloads: self-host Langfuse on your VPC (Docker Compose) so traces never leave your infra. Greenfield SaaS: start with hosted Langfuse cloud and migrate later if needed.
4. Always export OTelEven when you start with Langfuse SDK, configure it to also emit OTel-compatible spans. The day you swap or add a backend, the change is one config line, not a rewrite.

Four backends at a glance

Honeycomb and Datadog are the most common OTel backends — pick whichever your platform team already pays for.

Misconceptions

Same traces, stricter rules

Production agents in regulated industries (healthcare, finance, EU users) must meet strict logging requirements. Your traces and structured logs are the foundation — compliance adds rules about what you log, how you protect it, and how long you keep it.

Three frameworks dominate. HIPAA: log every access to protected health information, never log full patient records, six-year retention. SOC 2: every action attributable to a user, all auth events logged, ~one-year retention. GDPR: log the legal basis for processing, support right-to-erasure (delete all traces for one user on demand) — which means indexing every sink by user_id from day one.

The hospital chart, not the gossip log

BEFORE: A nurse writing patient notes on a whiteboard in the staff room — readable by anyone walking past, no signature, no timestamp, no access record.

PAIN: Rewind to the 1990s and that's how a lot of systems handled health data. One leak, one disgruntled employee, one stolen laptop and patient privacy is destroyed forever — with no audit trail to even prove what was taken.

MAPPING: A modern hospital chart is locked, signed, timestamped, and accessed only on a need-to-know basis. Every read is itself logged. Compliance logging is the agent equivalent: PII redacted before storage, access events themselves tracked, write-once / append-only storage so nothing can be quietly altered, retention windows tied to legal requirement (HIPAA 6yr, SOC 2 1yr).

Five compliance moves

1. Redact PII at the trace boundaryStrip emails, phones, SSNs, credit cards before the span reaches your logger. Before, never after — once raw PII hits the pipeline, scrubbing it from backups is painful.
2. Hash user identifiersDon't log user_id="alice@hospital.com". Log user_id="usr_8f14e45fce" — a one-way SHA-256 hash. You can still group traces by user; you can't reverse it.
3. Index by user_id from day oneGDPR right-to-erasure means "delete all traces for this user." Without a user_id index, that's a full-table scan over years of data. Index now or pay later.
4. Use append-only storage for audit logsS3 with Object Lock, or a dedicated audit-log service. Tamper-proof. Synchronised clocks. Separate from your application database so a compromised app can't rewrite history.
5. Apply framework-specific retentionHIPAA: 6 years. SOC 2: typically 1 year. GDPR: delete when the processing purpose expires — but keep anonymised audit records to prove you complied. Automate retention; don't rely on manual cleanup.

Pseudocode

The compliance pipeline: redact, hash, fan out to sinks with their own retention windows.

FUNCTION emit_span(span):
  # 1. REDACT before anything else
  span.input  = redact_pii(span.input)
  span.output = redact_pii(span.output)

  # 2. HASH user identifiers
  span.user_id = sha256(span.user_id)[:12]

  # 3. FAN OUT to sinks — each has its own retention
  audit_log.append(span)         # 6yr (HIPAA), immutable
  metrics.aggregate(span)        # 90d, no PII anywhere
  trace_store.write(span)        # 14d, debugging only

FUNCTION redact_pii(text):
  text = REPLACE email_regex   → "[EMAIL_REDACTED]"
  text = REPLACE phone_regex   → "[PHONE_REDACTED]"
  text = REPLACE ssn_regex     → "[SSN_REDACTED]"
  text = REPLACE cc_regex      → "[CC_REDACTED]"
  RETURN text

# GDPR right-to-erasure (handle on demand)
FUNCTION erase_user(user_id):
  hashed = sha256(user_id)[:12]
  FOR sink IN [audit_log, metrics, trace_store]:
    sink.delete_where(user_id=hashed)

Misconceptions

Treat prompts as code

Your agent's system prompt is as critical as any function in your codebase — a single word change can shift behaviour across thousands of requests. Yet many teams treat prompts as hardcoded strings buried inside application code, with no review process, no version history, and no way to roll back.

The discipline is simple: store prompts in version control, tag every trace with the prompt version that produced it, A/B test new versions before promoting, and run a regression eval suite on every prompt change. Prompt management closes the observability loop — once you can correlate behaviour to a specific prompt version, you can ship better prompts with the same confidence you ship better code.

The recipe vs the chef's notebook

BEFORE: A restaurant where the head chef keeps every recipe in their head — no written copy, no version history. The dish tastes great today; nobody can reproduce it tomorrow.

PAIN: The chef leaves. The new chef changes the salt by a pinch. Sales drop 20%. There's no way to know what changed, no way to roll back to last month's recipe, no way to A/B the new version against the old.

MAPPING: A serious kitchen keeps recipes in a binder, with version numbers, change logs, and a tasting panel that signs off on every revision before it hits the menu. Prompts are recipes. Git is the binder. Eval suites are the tasting panel. Once your prompt_version is in every trace, you can dashboard "task success rate by prompt version" and prove a change before promoting it.

Three pillars of prompt management

1. Version controlStore every system prompt, few-shot example set, and tool description in files like prompts/order_agent_v12.txt. Git history shows when, who, and why. A prompt registry (config file or Langfuse prompt mgmt) maps names to versions at runtime: get_prompt("order_agent", "latest").
2. A/B testingDon't replace the old prompt — route 5–10% of traffic to the new version. Compare metrics: task success, tokens used, satisfaction, hallucination rate. Tag every trace with prompt_version so dashboards slice cleanly. Promote to 100% only when the new version proves at least as good.
3. Regression testingMaintain a 20–50-input eval suite with expected outputs. Run it automatically (CI / pre-merge) on every prompt change. If accuracy drops below threshold, block the merge. Catches the prompt tweak that fixes one edge case but degrades ten common ones.

Combine with the eval framework from M18 for a complete quality gate — prompts can no longer ship without proof.

Should I ship this prompt change?

# New prompt version proposed. What now?

IF change is just a typo / formatting fix:
  RUN regression eval → IF pass: ship 100%
  # still tag with new prompt_version

ELIF change reshapes behaviour (instructions, tools, format):
  RUN regression eval (block on failure)
  ROUTE 5-10% of traffic to new version
  DASHBOARD metrics by prompt_version FOR 24-72h:
    - task_success_rate
    - tokens_per_request
    - hallucination_flags
    - user_satisfaction
  IF new ≥ old ON ALL metrics:
    PROMOTE to 100%
  ELSE:
    ROLL BACK — investigate the regression

ELIF change is for one customer / one edge case:
  CONSIDER per-tenant prompt variant instead
  # avoid degrading the 99% to fix the 1%

ELSE:
  DON'T SHIP. If the change has no metric story, the
  next "small tweak" will silently undo it.

Misconceptions

One question per concept

Tap a card to reveal the answer.