The 5 concepts of structured output

Tap any concept to jump to its 5-card cluster.

• 1Why Agents Need Structured ResponsesData for code, not prose for humans
• 2Tool Use as Structured OutputThe order form the model must fill in
• 3Validation, Stop Sequences & Schema CheckingDefense in depth: parse, then verify
• 4Error Recovery: When Parsing FailsRetry with the error, don't give up
• 5Multi-Modal Input: Vision and PDFAgents that read images and documents

Agents produce data, not prose

An agent's output rarely stops at your screen. It gets parsed by a JSON deserializer, written to a database, or sent to an API — and every one of those steps expects predictable shapes: labeled fields, typed values.

Free-form text can't do that reliably. "John works at Acme, email maybe john@acme.com" forces fragile regex and guesswork. Structured data is a filled-in spreadsheet instead of a paragraph: every field labeled, every value typed, grabbed with zero guessing.

Spreadsheet, not a rambling coworker

BEFORE: Imagine asking a coworker for customer data and they reply with a ramble: "Oh yeah, John's over at Acme, his email's something like john@acme.com, and he might be a PM?"

PAIN: You'd have to read it, guess which words are data, and copy them out by hand — every single time. Now do that hundreds of times a minute in code, where one missed field crashes the pipeline at 2 AM.

MAPPING: Structured data is getting a filled-in spreadsheet instead of that paragraph. Every field has a label, every value has a type, and your code grabs exactly what it needs — no reading, no guessing, no 2 AM page.

From text to something code can use

1. Model reads messy inputAn email signature, a note, a paragraph — natural language written for humans.
2. It emits labeled fieldsInstead of prose, it returns name, email, company — each with a value and a type.
3. Your code consumes it directlyDeserialize the JSON, insert to a DB, call an API — no string parsing in between.
4. Downstream stays stableBecause the shape is predictable, one weird input can't silently break the whole flow.

Pseudocode

raw = "John is a PM at Acme, email john@acme.com"

# Unstructured path — fragile:
data = parse_json(raw)     # SyntaxError!
# it was prose, not JSON -> pipeline crashes

# Structured path — code-ready:
contact = {
  name:    "John",
  role:    "PM",
  company: "Acme",
  email:   "john@acme.com",
}
db.insert(contact)         # just works

Misconceptions

Tool use is your reliable structure lever

The most reliable way to get structured output isn't "please reply in JSON" — it's tool use. You define a tool with a JSON Schema for its parameters, and the model returns a tool_calls entry: a tool name plus arguments matching your schema.

The twist: you don't have to actually run the tool. Define one purely as a shape, force the model to call it, and just read the values. Because the model is trained on the tool-calling format, it fills the form instead of writing loose text.

A pre-printed order form

BEFORE: Without tool use, getting structured output was like shouting your order across a noisy room — you'd write "please return JSON with these fields" and hope the model heard you right.

PAIN: It might wrap the JSON in markdown, forget a required field, use the wrong type, or slip a friendly sentence in front of the data. You'd need fragile regex and try/catch just to dig the values out.

MAPPING: Tool use hands the model a restaurant order form with pre-printed blanks: dish, quantity, special instructions. It can only fill in the blanks — when forced to use a tool, it physically can't return free-form text. The form (the schema) guarantees the structure.

Tool use in four moves

1. Define the toolGive it a name, a description, and a JSON Schema for its parameters — field names, types, which are required.
2. Force the callSet tool_choice to that specific tool. Now a structured tool call is guaranteed, not hoped for.
3. Read the tool callThe reply has no text — the data lives in tool_calls[0].function.arguments as a JSON string.
4. Parse the argumentsRun json.loads() on that string to get a real object your code can use.

Pseudocode

tool = {
  name: "extract_contact",
  params_schema: {
    name:  string  (required),
    email: string  (required),
    phone: string  (optional),
  }
}

reply = model.call(
  text,
  tools = [tool],
  force_tool = "extract_contact",  # tool_choice
)

call = reply.tool_calls[0]
data = parse_json(call.arguments)  # the fields!

Misconceptions

Defense in depth for output

Reliable output stacks three layers so a miss in one is caught by the next. Layer 1: format constraints (tool use) shape valid JSON up front. Layer 2: stop sequences halt generation at the closing brace so no chatty text trails your object.

Layer 3: schema validation with Pydantic (Python) or Zod (TypeScript). Plain json.loads() only checks syntax — matching braces. A schema checks semantics: does every required field exist, is every type right, did junk sneak in? With all three, you approach near-100% valid output.

The director yells "Cut!", then the inspector checks

BEFORE: Without validation layers you'd send a prompt, get back some JSON-ish text, cross your fingers, and call parse(). Sometimes clean, sometimes prefixed with "Here's the data:" that broke everything.

PAIN: In production that meant 5–15% of responses failed to parse — silent data loss or crashes that surfaced hours later. Intermittent and format-dependent, so debugging was a nightmare.

MAPPING: A stop sequence is a film director yelling "Cut!" at exactly the closing brace — generation halts, no trailing lines. Schema validation is the quality inspector checking every frame after the cut. Together they're a multi-layer safety net.

The three layers

1. Format constraintsTool use (or careful prompting) makes the model emit valid JSON in the right shape from the start.
2. Stop sequencesStrings like } tell the API to halt the moment they appear, so no conversational text follows the object.
3. Schema validationPydantic/Zod check required fields, correct types, and no surprise data — the semantic pass.
4. On failure, raise detailsA ValidationError names the exact field ("email expected str, got None") — fuel for the retry prompt.

Which layer catches what?

IF output has chatty text around JSON:
  → add a stop sequence (or use tool use)

IF braces / commas are malformed:
  → json parse (syntax) catches it

IF a required field is missing / wrong type:
  → schema validation catches it

IF value is valid-shaped but nonsense
   (age = 350):
  → needs YOUR business-rule check

# json.loads = syntax; schema = semantics

Trick: auto-generate the tool schema from your Pydantic model so the two never drift apart.

Misconceptions

Recover, don't crash

No matter how good your schema is, parse failures will happen in production. The question isn't "if" but "how do you respond?" The best move is retry with error feedback: send the request again, but append the specific error so the model self-corrects. That fixes ~90% on the first retry.

Beyond that: fall back to a simpler schema, partial-parse the fields that did validate, try cascading validators, or escalate to a human. Across all of them, use exponential backoff (1s, 2s, 4s) and a max retry count so you never loop forever burning tokens.

GPS recalculating, not stalling

BEFORE: Early integrations treated a parse failure as fatal — malformed JSON, request dead, user sees "Something went wrong."

PAIN: One missing comma wasted the whole API call, left a customer-facing request unanswered, and needed a human to unblock the pipeline. Expensive for a stray character.

MAPPING: Error recovery is a GPS recalculating your route. Miss a turn (a parse failure) and it doesn't pull over and kill the engine — it instantly recomputes, tells you exactly which turn you missed, and offers a corrected path to the same destination.

The recovery ladder

1. Retry with the errorResend, appending the exact failure ("email was null, expected string"). Fixes ~90% on the first retry.
2. Fall back / partial-parseAsk for a simpler schema, or keep the fields that did validate and flag the rest. Some data beats none.
3. Back off & capDouble the wait each time (1s, 2s, 4s) and stop after ~3 tries so a gibberish input can't loop forever.
4. Escalate to a humanWhen every automated path fails, route to a reviewer — your safety net for truly ambiguous input.

Pseudocode

last_error = none

FOR attempt IN 1..max_retries:
  prompt = base_prompt
  IF last_error:
    prompt += "Previous attempt failed: "
    prompt += last_error        # be specific!

  TRY:
    data = model.call(prompt, force_tool)
    RETURN validate(data)         # success
  CATCH ValidationError AS e:
    last_error = e.details
    wait(2 ** attempt)          # backoff

RAISE "gave up after max_retries"

Misconceptions

Agents that see, not just read

Agents don't only consume text. A capable model can process images (JPEG, PNG) and PDFs directly — so an agent can read a scanned invoice, pull data from a photo of a shipping label, or interpret a chart, no separate OCR pipeline required.

You send an image as a base64 string in the message's content array with type: "image" and a media_type. PDFs go in natively for multi-page documents. Combine multi-modal input with tool use and you extract structured data straight from a picture.

An assistant who can look at the paper

BEFORE: A text-only agent is a assistant you can only talk to over the phone — you have to describe every document out loud before they can help.

PAIN: Most real business input is scanned filings, photographed receipts, and PDF contracts. Reading each one aloud (or running a brittle OCR step first) is slow, lossy, and misses layout and tables entirely.

MAPPING: Multi-modal input is handing that assistant the actual sheet of paper. They look at the scanned filing, read the debtor name and filing date off the page, and hand you back a clean structured record — no dictation, no separate OCR service.

Sending an image or PDF

1. Encode the fileTurn the image into a base64 string (PDFs can be sent natively for multi-page docs).
2. Build a content arrayAdd an image block with the base64 data and media_type, plus a text block with your ask.
3. Attach a toolInclude your structured schema so the model returns fields, not a paragraph.
4. Parse & validateRead the tool call, validate against the schema, return a clean structured record.

Pseudocode

TOOL analyze_document(doc_base64, media_type):

  reply = model.call(
    content = [
      { type: "image",
        source: doc_base64,
        media_type: media_type },
      { type: "text",
        text: "Extract filing number,"
            + " debtor, date. Return JSON." },
    ],
    tools = [ filing_schema ],
  )

  data = parse_json(reply.tool_calls[0])
  RETURN validate(data)   # structured record

Misconceptions

Test the core insight

One question per concept. Tap to reveal the answer.