The 7 concepts of agent observability

Tap any concept to jump to its 5-card cluster.

• 1Why Observability MattersAnswer any question after the fact
• 2Agent TracesThe flight recorder for one request
• 3Spans: Nesting & TimingWhere the time actually goes
• 4Structured LoggingA filing cabinet, not a junk drawer
• 5Observability ToolsLangfuse, OpenTelemetry, LangSmith, Phoenix
• 6Compliance & Audit LoggingHIPAA, SOC 2, GDPR
• 7Prompt Management & VersioningTreat prompts as code

See why it broke, not just that it broke

Observability means instrumenting a system so you can answer questions about its behaviour after the fact, without deploying new code. Monitoring tells you that something broke; observability tells you why.

Agents need it more than most software, for three reasons. They’re non-deterministic, so you can’t reliably reproduce a bug. They make 5–20 decisions per request, so a bug in step 7 may only surface in step 12. And every LLM call costs money, so a runaway loop can burn a budget unseen.

Driving at night without instruments

BEFORE: Picture driving at night with no headlights, no dashboard gauges and no GPS. You press the accelerator and hope.

PAIN: When something goes wrong you have no idea why until you crash. Running an agent without observability is the same: a user reports a wrong answer and you have no record of the tools called or where the chain broke.

MAPPING: Observability is headlights, dashboard and GPS combined. Traces show the road the agent took, spans time each turn, and structured logs are the gauges you can read afterwards.

The three pillars

1. TracesA structured record of one whole request: every step, in order.
2. LogsMachine-parseable events: one JSON line per decision, with a trace_id.
3. MetricsAggregates over many requests: latency percentiles, cost, error rate.
4. TogetherQuestions you didn’t anticipate when writing the code become answerable.

What should I record?

RECORD:
  every decision, tool call and LLM call
  timing (latency_ms) per step
  tokens in / out, and cost
  errors and stop reasons
  a trace_id on every line

DO NOT RECORD:
  everything -- noise and cost
  PII, secrets, full prompts

GOAL: any bug answerable from
  the record, without reproducing it

Teams without tracing spend an average of 4–8 hours per agent failure; with it, 10–15 minutes.

Misconceptions

One trace, one request, the whole story

A trace is a complete record of a single agent execution. It has four parts: a unique trace_id, start and end timestamps, request metadata (user_id, session_id, an input summary), and a tree of spans — the individual units of work.

Traces follow the OpenTelemetry standard, so the same instrumentation works across languages and ships to any compatible backend: Langfuse, Jaeger, Datadog and others.

The flight recorder

BEFORE: After a safe landing nobody needs the flight recorder. After a near-miss, it’s the most valuable evidence there is: every input and every response on one timeline.

PAIN: Without it, investigators rely on pilot memory. For agents, “pilot memory” is logs that say “request received” and “response sent” with nothing in between.

MAPPING: A trace is the agent’s flight recorder: every decision, tool call and model call from the moment a request arrives to the response, retrievable by trace_id months later.

What a trace captures

1. InputThe user’s message — or a hash or summary if it contains PII.
2. Model and tool callsWhich model or tool, inputs, tokens, latency, success or failure.
3. DecisionsWhat the agent chose at each step, and why.
4. Output and metadataThe final answer, total cost, total latency and error status.

A trace object

trace = {
  trace_id:    "tr_a1b2c3d4e5f6",
  name:        "order-agent",
  duration_ms: 1607,
  user_id:     "usr_8f14e45fce", # hashed
  input:  "Where is my order ORD-12345?",
  output: "Shipped via FedEx...",
  status: "ok",
  spans: [ llm.chat 847ms,
           tool.search_orders 47ms,
           llm.chat 712ms ]
}

One JSON object tells the whole story: who asked, what the agent did in three spans, how long each took, and what it answered.

Misconceptions

Spans show where the time went

A span is one unit of work inside a trace. It has a span_id, a parent_span_id linking it to the span that triggered it, a name, start and end times, a status, and attributes such as model and token counts.

The parent links form a tree, drawn as a waterfall. Five span types cover an agent: the root (whole request), LLM calls, tool calls, retrieval lookups and guardrail checks.

Timing every step of a recipe

BEFORE: “Make lasagna” contains “make sauce” (chop, brown, simmer), “make pasta layers” and “assemble and bake” — each nested inside a bigger step.

PAIN: If it tastes wrong, you need to know which step failed. Without timing each one, you taste the final dish and guess backwards.

MAPPING: The root span is “make lasagna”; LLM, tool and guardrail calls are its sub-steps, each timed. The nesting shows exactly where the time and the errors come from.

Three questions spans answer

1. Where is time spent?Is it LLM calls, or one tool taking 4 seconds?
2. Any sequential bottlenecks?Two independent lookups back to back could run in parallel for a 2× speedup.
3. A slow dependency?A database call at 50 ms last week and 2000 ms today means the database changed, not your agent.
4. Don’t assumeThe LLM is often not the bottleneck; databases and third-party APIs frequently are.

Nested spans

root = START SPAN "agent.run"
  WHILE agent not done:
    llm = START SPAN "llm.chat" UNDER root
      reply = ASK Claude(...)
      llm.set(model, tokens_in, tokens_out)
    END llm
    FOR each tool call IN reply:
      t = START SPAN "tool."+name UNDER root
        result = RUN tool
        t.set(success, latency_ms)
      END t
END root  # the waterfall is this tree

At 10,000 requests a day, finding that 40% of time sits in one cacheable tool call can cut P50 latency from 3s to 1.8s.

Misconceptions

Every log line is queryable JSON

Structured logging writes each event as a JSON object with consistent keys — trace_id, span_id, timestamp, level, tool_name, latency_ms — instead of free-form text.

That makes logs queryable: “every ERROR where tool_name is search_orders and latency_ms > 2000” runs in seconds across millions of lines, no regex needed. Equally important is what you never log: PII, secrets, full prompts and full responses.

The trade-off is deliberate. A text line like “search_orders took 47ms, found 3” is easy to read in a terminal but impossible to query at scale; the JSON version is slightly harder to read and trivial to filter, aggregate and alert on.

Filing cabinet versus junk drawer

BEFORE: You can toss every receipt into one drawer, or file each in a labelled folder by date, category and amount.

PAIN: Most logs are the junk drawer: “INFO: Processing request…”. Finding the one failure at 3:47 AM among 10 million lines means fragile regex and false matches.

MAPPING: Structured logs are the filing cabinet. Every entry has the same labelled fields, so you pull exactly the folder you need.

A redacted logging pipeline

1. Log the eventlogger.info("tool_call", tool_name=…, latency_ms=47).
2. Add automatic fieldsTimestamp, level and process ID are added for you.
3. Redact firstEmails, names and IDs are stripped or hashed before anything is stored.
4. Fan outOne clean JSON line feeds the aggregator, alerts and dashboards.

One structured log entry

LOG {
  trace_id:  "tr_a1b2c3d4e5f6",
  span_id:   "sp_7g8h9i0j",
  timestamp: "2025-03-15T14:32:01.847Z",
  level:     "info",
  agent_id:  "order-agent-v2",
  model:     "claude-sonnet-4-6",
  tokens:    { in: 1840, out: 312 },
  tool_name: "search_orders",
  latency_ms: 847,
  user_id:   "usr_hashed_abc"  # never raw
}

Never log PII, API keys, authorization headers, full prompts, full responses, medical data or card numbers. Keep full prompt/response pairs only in an access-controlled trace store.

Misconceptions

Pick the tool for your stack

Langfuse is open source and LLM-native: it understands tokens, model cost and prompts, and can be self-hosted. OpenTelemetry is the vendor-neutral CNCF standard: instrument once and ship to Datadog, Grafana or New Relic alongside the rest of your stack.

LangSmith fits teams on LangChain; Arize Phoenix focuses on embedding drift and RAG quality. Many production teams run OpenTelemetry for platform monitoring and Langfuse for the LLM-specific workflow.

The specialist and the hospital system

BEFORE: A clinic can buy a specialist cardiology monitor, or plug into the hospital-wide system every other ward already uses.

PAIN: The specialist monitor shows heart detail the general system doesn’t, but it lives apart from everything else. The hospital system connects to every ward but doesn’t know cardiology out of the box.

MAPPING: Langfuse is the specialist: prompts, costs and traces for LLMs. OpenTelemetry is the hospital system: your agent’s spans sit in the same waterfall as your API server’s. Plenty of teams use both.

Choosing

1. Building directly on Claude?Langfuse or OpenTelemetry are the strongest choices.
2. Need agent-specific UX?Langfuse: prompt playground, cost rollups, trace diffing.
3. Platform already on an APM?OpenTelemetry: SLO dashboards and on-call alerting in one place.
4. On LangChain, or RAG-heavy?LangSmith for LangChain; Phoenix for embedding drift and retrieval quality.

Tracing that never breaks the agent

TRY: tracer = CONNECT Langfuse(keys FROM env)
ON error: tracer = NONE  # agent still runs

ON each request:
  TRY: span = tracer.start("llm.call")
  ON error: span = NONE
  reply = ASK Claude(...)   # the real work
  TRY: span.end(tokens, latency_ms)
  ON error: LOG "trace dropped" to stderr

# Langfuse down at 3am? Lose 30 minutes of
# traces, not 1,000 users. Never the reverse.

The desktop walkthrough builds this in full: an order-tracking agent with Langfuse tracing, structlog or pino logging, and PII redaction.

Misconceptions

What regulated agents must log — and keep

Regulated agents add rules about what you log, how you protect it and how long you keep it. HIPAA: log every access to health information — who, which tool, which record, when — with record IDs, not full records. SOC 2: log authentication, authorization decisions and configuration changes, attributable to a user.

GDPR: log the legal basis for processing, and be able to find and delete every trace for a given user — which means indexing every store by user_id from day one.

The bank vault’s visitor log

BEFORE: A bank vault keeps a visitor log: who entered, when, which box they opened.

PAIN: If that log could be edited afterwards, or recorded what was inside each box, it would be useless as evidence and a liability of its own.

MAPPING: An audit log is the vault’s visitor log: it records access, not contents, and it can’t be changed. Redaction keeps the contents out; append-only storage keeps it honest.

Redact, then store

1. Redact before loggingMask emails, phones, SSNs and card numbers before data reaches any pipeline.
2. Immutable audit logAppend-only storage such as S3 with Object Lock, separate from the app database.
3. Retention per frameworkHIPAA 6 years, SOC 2 typically 1 year; GDPR deletes when the purpose expires.
4. Erasure by user_idIndex every sink by user_id so a deletion request is one query.

What goes where

AUDIT LOG (immutable):
  who accessed what, when, via which tool
  authorization decisions, config changes
  retention: HIPAA 6y, SOC 2 ~1y

TRACE STORE (debugging):
  redacted spans, short retention (~14d)

METRICS (aggregates, ~90d)

EVERY SINK: indexed by user_id
  # GDPR erasure = one delete query

Retrofitting deletion across years of unindexed traces is painful. Build retention automation early.

Misconceptions

Prompts are code: version, test, roll out

A single word in a system prompt can change behaviour across thousands of requests, yet prompts often live as hard-coded strings with no history. Treat them as code: store each prompt in a version-controlled file and load it from a registry at runtime.

Roll changes out gradually. Send 5–10% of traffic to the new version, log prompt_version on every trace, and compare success rate, tokens and hallucination rate. Run a regression suite on every prompt change, too.

A recipe change in a restaurant chain

BEFORE: A restaurant chain’s head chef tweaks the house sauce and emails the new recipe to every kitchen that night.

PAIN: Next morning 200 kitchens serve it at once. If it’s worse, nobody knows which version a complaint was about or how to go back.

MAPPING: Prompt management is how the chain should do it: numbered recipe versions, a trial in a few kitchens, the version noted on every order, and a switch back if the numbers drop.

Three pillars

1. Version controlprompts/order_agent_v12.txt in Git: who changed what, when and why.
2. Registryget_prompt("order_agent", version="latest") switches versions without a redeploy.
3. A/B rollout5–10% of traffic first; promote to 100% only once metrics hold.
4. Regression gate20–50 representative cases run whenever a prompt file changes.

A gated prompt change

new = LOAD prompt "order_agent" v13
ON prompt file change:
  scores = RUN eval suite (20-50 cases)
  IF scores < baseline: BLOCK merge

ROUTE 10% of traffic to v13
TAG every trace with prompt_version
COMPARE v12 vs v13:
  success rate, tokens, hallucinations
IF v13 ≥ v12 on every metric:
  PROMOTE v13 to 100%
ELSE: ROLL BACK to v12

Combine this with the evaluation framework from M18 for a complete quality gate on prompts.

Misconceptions

Test the core insight

5 questions on the module’s sharpest insights. Tap to reveal the answer.