The 5 concepts of tracing

Tap any concept to jump to its 5-card cluster.

• 1Why Tracing MattersThe flight data recorder
• 2The Four CategoriesTools, LLM turns, loops, errors
• 3Structured LoggingEvery line is queryable JSON
• 4Distributed TracingOpenTelemetry + Jaeger spans
• 5Custom TracerZero-dependency @trace_agent

You can't fix what you can't replay

A trace is the complete record of one agent run: every LLM call, every tool invocation, every loop iteration, every error — timestamped and linked into a causal chain. It's the difference between "something felt off" and "the search_filings tool timed out on turn 3".

Teams with structured traces report fixing bugs 10x faster than teams debugging with print(). The trace eliminates the two most expensive activities: reproducing the failure (you have the exact inputs) and finding which component broke (you have per-step latency, not one wall-clock total).

The flight data recorder

BEFORE: Commercial aircraft carry two black boxes that record every sensor reading — altitude, thrust, control angles, cockpit audio — 25 times per second, always running, whether or not anything goes wrong.

PAIN: Imagine instead that pilots just described the crash from memory: "something felt off around 30,000 feet and then everything went sideways." Investigators couldn't tell instrument failure from pilot error. Every investigation starts from zero. That's debugging an untraced agent.

MAPPING: Your structured trace IS the black box. Every LLM call, tool call, token count, and latency is a sensor reading written to durable storage the moment it happens. When something fails, you replay the recording frame-by-frame and find the exact wrong decision.

From symptom to root cause

1. Capture as it happensEvery event is written to storage the moment it occurs — not reconstructed later from memory.
2. Link by run_idAll events from one run share an ID, so you can pull the full causal chain for a single failure.
3. Filter to the failureQuery exit_reason: "tool_error" and jump straight to the failing call with its exact arguments.
4. Replay & fixYou have the inputs and per-step timings, so a 4-hour hunt becomes a 20-minute fix.

Trace or print()?

IF the agent runs unattended
   (batch jobs, production, 2AM):
  → structured tracing is mandatory

IF failures are hard to reproduce
   (nondeterministic LLM output):
  → you need captured inputs — trace

IF it's a one-off script you watch
   run once in a terminal:
  → print() is fine

# rule: if you'd cry finding the bug
# by hand, trace it up front.

Misconceptions

Every agent event is one of four kinds

There are exactly four trace categories in any agent run, and each has its own relevant fields: tool calls (name, args, output, latency), LLM turns (token counts, model, finish reason), loop iterations (iteration number, tools used, exit reason), and errors (exception type, stack tail, retry info).

Capture all four and you have a complete causal picture of any run. Miss one and you get a blind spot exactly where you need visibility most — usually the error path.

A hospital chart with four vitals

BEFORE: A nurse who records only "patient seems fine" gives the next doctor nothing to act on. Useful charts track specific vitals: heart rate, blood pressure, temperature, oxygen.

PAIN: If you record vitals in a different format every shift — sometimes prose, sometimes numbers, sometimes nothing — no one can compare readings or spot a trend. The chart becomes noise.

MAPPING: The four trace categories are your four vitals. Every event slots into tool call, LLM turn, loop iteration, or error, each with known fields. Now any run reads like a clean chart: consistent, comparable, and complete.

The four categories in detail

1. Tool callsname, args, output (truncated ~2 KB), latency, success flag, error message if any.
2. LLM turnsprompt & completion tokens, model name, finish reason (stop / tool_calls / length / error), latency.
3. Loop iterationsiteration number, tools invoked this turn, exit reason (max_iterations / goal_achieved / tool_error), wall time.
4. Errorsexception type, message, last 3 stack frames, whether a retry was attempted and how many.

Pseudocode

# ONE event shape covers all four kinds
DEFINE TraceEvent:
  category   # tool_call|llm_turn|loop_iter|error
  run_id     # links all events in one run
  ts         # when it happened
  latency_ms # shared by most events
  # + fields specific to each category

FUNCTION emit(event):
  IF event.category NOT IN valid_four:
    raise error
  append event AS json_line TO file

One schema means one destination, one filter syntax, one replay format — not four scattered logging patterns.

Misconceptions

Every log line is machine-readable JSON

Structured logging means each log event is a valid JSON object with consistent key-value fields — timestamp, level, message, plus your custom fields — not a free-text string like "Tool called: fuzzy_match with args ['Acme','ACME']".

The payoff: you can query logs with standard tools. cat agent.log | jq 'select(.latency_ms > 2000)' finds every slow step across millions of lines with zero parsing code. Libraries like structlog (Python) and pino (Node) add the timestamps and levels for you.

Spreadsheet vs. sticky notes

BEFORE: Picture tracking your expenses on a wall of sticky notes — each in your own words: "coffee-ish, maybe $4?", "lunch downtown Tuesday".

PAIN: To answer "how much on food last month?" you'd re-read and reinterpret every note by hand. There's no way to sort, filter, or sum — the data is trapped in prose.

MAPPING: Structured logging is the spreadsheet: every entry has the same columns (date, category, amount). Now "sum food in March" is one formula. JSON fields are those columns — jq and Pandas are your formulas.

Wiring up structlog / pino

1. Configure once at startupSet up JSON output with timestamp + level processors. Call it exactly once — twice duplicates processors.
2. Bind the run_idAttach the run ID to the logger so every event in the run carries it automatically.
3. Emit one event per actionWrap each LLM call and tool call; log fields on success AND before re-raising on error.
4. Ship the JSON linesWrite to a file and/or a backend. Each line is independent, so appends are cheap and atomic.

Pseudocode

CONFIGURE logger ONCE:
  output = JSON
  add timestamp, level automatically

log = logger.bind(run_id)

FUNCTION llm_call(messages):
  t0 = now()
  TRY:
    reply = model.chat(messages)
    log.info(event="llm_turn",
      tokens=reply.usage,
      latency=now() - t0)
    RETURN reply
  CATCH err:
    log.error(event="error", msg=err)
    RAISE   # log BEFORE re-raising

Misconceptions

Spans nest into a visual call tree

OpenTelemetry (OTEL) is the industry-standard, vendor-neutral framework for distributed tracing. Your agent creates a root span — one named, timed operation — for each run, then child spans for each LLM call and tool call. Parent-child nesting encodes causality.

An exporter ships those spans over the wire (OTLP/gRPC) to a backend like Jaeger, which renders them as a nested waterfall you can filter by service, operation, or latency. It runs fully locally — no data leaves your machine. (Cloud alternative: LangSmith's @traceable, which works with your Ollama client unchanged.)

A project Gantt chart

BEFORE: A manager who only knows "the project took 6 weeks" can't tell which phase dragged. They need to see each task's start, end, and how tasks nest under phases.

PAIN: Without that breakdown, every delay looks the same. You can't tell if design ran long or if one vendor blocked everything downstream. You just have a total.

MAPPING: Spans are tasks on a Gantt chart. The root span is the whole project; child spans are phases and sub-tasks, drawn as nested bars on a timeline. One glance shows the 823 ms tool call that's the real bottleneck — not the 3,241 ms total.

From agent loop to Jaeger UI

1. Bootstrap onceWire a TracerProvider to an OTLP exporter pointed at Jaeger (localhost:4317). A batch processor flushes with <1 ms overhead.
2. Open a root spanStart a span for the whole run; set attributes like query and exit reason.
3. Nest child spansEach LLM call and tool call opens its own span with model, tokens, and latency attributes.
4. End in finallyAlways close spans in a finally block — an escaped exception orphans the span and it never exports.

LangSmith or OpenTelemetry?

IF you want the fastest setup
   and don't mind cloud hosting:
  → LangSmith @traceable
     (5k free traces/month)

IF data must stay on-machine
   or you span multiple services:
  → OpenTelemetry + local Jaeger

IF you need vendor-neutral,
   production-grade standard:
  → OpenTelemetry (OTLP export)

# both work with Ollama unchanged

Misconceptions

Tracing with zero dependencies

You don't always need a platform. A @trace_agent decorator wraps any agent function and writes run_start, sub-events, and run_end to a JSON Lines file — using only the standard library. Drop it into any project in 30 seconds, no pip install.

JSON Lines (JSONL) is one complete JSON object per line, no commas between them. Because each line is independent, you append without reading the file back — ideal for streaming logs. To read a run, open the file and parse each line, then replay() it into a summary.

A pocket notebook vs. a CRM

BEFORE: You could log every customer interaction in an enterprise CRM — powerful, but it takes setup, licenses, and onboarding before you write a single note.

PAIN: For a solo shop or a quick experiment, that's overkill. By the time the CRM is configured, you've lost the notes you wanted to take today.

MAPPING: The custom tracer is a pocket notebook: no setup, always in your pocket, one line per entry. It won't replace a CRM at scale, but it captures everything right now — and JSONL means you can import those notes into the "CRM" (Pandas, jq) later.

What the decorator does

1. Mint a run_id & fileOn each call, generate a short ID and open a per-run trace_<id>.jsonl file.
2. Emit run_startRecord the function name and (truncated) arguments before the work begins.
3. Inject an emit hookIf the function accepts an _emit param, pass it in so it can log sub-events like LLM turns.
4. Emit run_endOn success OR exception, write the outcome, total latency, and error info — then re-raise.

Pseudocode

DECORATOR trace_agent(fn):
  RETURN FUNCTION wrapper(args):
    run_id = short_uuid()
    file   = "trace_" + run_id + ".jsonl"

    FUNCTION emit(event):
      event.run_id = run_id
      event.ts = now()
      append json(event) TO file

    emit({start, fn.name, args})
    IF fn accepts _emit: pass emit in
    TRY:
      result = fn(args)
      emit({end, "success", latency})
      RETURN result
    CATCH err:
      emit({end, "error", err})
      RAISE

Misconceptions

Test the core insight

One question per concept. Tap to reveal the answer.