M04: Structured Output & Parsing

Agents don't just generate text — they produce data that your code must parse, validate, and act on. This module teaches you how to get reliable, structured output from the model and what to do when it isn't perfect.

Learning Objectives

• Explain why agents require structured output instead of free-form text
• Use the model's tool use feature to guarantee structured JSON responses
• Validate API responses with Pydantic (Python) and Zod (TypeScript) schemas
• Implement retry logic with error-aware re-prompting for parse failures
• Build a complete data extraction pipeline with schema validation and error recovery

Why Agents Need Structured Responses

What this actually looks like: Here's the difference between unstructured and structured output from the same prompt. The left side is what you'd get from a plain text response. The right side is what tool use gives you:

# Unstructured (free-form text — hard to parse reliably):
"Jane Smith is the VP of Engineering at TechCorp. Her email is
 jane.smith@techcorp.io and you can reach her at (555) 123-4567."

# Structured (tool call response — code-ready):
{
  "name": "Jane Smith",
  "email": "jane.smith@techcorp.io",
  "phone": "(555) 123-4567",
  "company": "TechCorp",
  "role": "VP of Engineering"
}

Tool Use as Structured Output

What this actually looks like in the API response: When the model uses a tool, you don't get a plain text string. You get a structured tool_calls array on the response message:

# The model's actual response when forced to use a tool:
{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": null,
      "tool_calls": [{                  # ← not "content" text!
        "id": "call_a1b2c3d4",
        "type": "function",
        "function": {
          "name": "extract_contact",    # ← which tool it "called"
          "arguments": "{\"name\": \"Jane Smith\", \"email\": \"jane.smith@techcorp.io\", \"phone\": \"(555) 123-4567\", \"company\": \"TechCorp\", \"role\": \"VP of Engineering\"}"
        }                               # ← arguments is a JSON *string* — json.loads() it
      }]
    },
    "finish_reason": "tool_calls"        # ← stopped because it wants to call a tool
  }]
}

Validation, Stop Sequences & Schema Checking

Schema Validation with Pydantic & Zod

PydanticA Python library for data validation using type annotations. You define a class with typed fields, and Pydantic validates, parses, and transforms incoming data automatically. The standard tool for validating LLM output in Python agents. (Python) and ZodA TypeScript-first schema validation library. You define schemas with z.object() and Zod validates data at runtime, providing detailed error messages for each invalid field. The TypeScript equivalent of Pydantic. (TypeScript) are schema validation libraries. In plain English: you define the exact shape your data must have (which fields, what types, which are required), and the library automatically checks every response against that shape. If something doesn't match, it tells you exactly what's wrong.

Under the hood, these libraries work by defining a model class (Pydantic) or schema object (Zod) with typed fields. When you pass the model's output through the model, the library checks each field: Is name a string? Is price a number? Is email present (since it's required)? If any check fails, it raises a ValidationErrorAn exception raised by Pydantic when input data fails validation. It contains a list of specific field errors — which field failed, what type was expected, and what was received — making it ideal for retry prompts. with field-level details — not just "invalid data" but "field 'email' expected str, got None." These specific error messages are exactly what you'll feed back into retry prompts.

How does this differ from just calling JSON.parse() or json.loads()? Those only check that the JSON is syntactically valid — matching braces, correct commas. Schema validation goes further: it verifies that the data is semantically correct for your application. Valid JSON like {"name": 123} would pass JSON.parse() but fail Pydantic because name should be a string, not a number. The tool's parameters are specified in the parametersA field inside each tool definition's function object that contains a JSON Schema describing the tool's expected parameters — their names, types, descriptions, and which are required. the model uses this schema to generate valid arguments. field of the tool definition. Here's a time-saving trick: you can auto-generate this schema from your Pydantic model using ContactInfo.model_json_schema(). That way, you define the shape once in Pydantic, and the tool definition stays in sync automatically.

Error Recovery: When Parsing Fails

Code Walkthrough: Data Extraction Pipeline

Approach 1: Tool Use for Structured Output

Let's start with the data model. The code below defines a ContactInfo schema in two places: a Pydantic model (your validation blueprint) and a matching tool definition (what the model sees). Having both gives you a two-layer guarantee — the model's tool call ensures the right structure, and Pydantic ensures the right types and values. One important gotcha: these two definitions must stay in sync. If you add a field to one, add it to the other. In production, you'd use ContactInfo.model_json_schema() to auto-generate the tool schema from Pydantic, eliminating the sync problem entirely.

The interesting part is the extract_contact() function. It sends text to the model with tool_choice={"type": "function", "function": {"name": "extract_contact"}} — this is the critical line that forces the model to return a structured tool call instead of free-form text. The function then reads the first entry in response.choices[0].message.tool_calls, parses its function.arguments string with json.loads, and validates the result through Pydantic. Here's the important nuance: even with forced tool use, the values inside the JSON might be wrong (the model could hallucinate an email, for example). Tool use guarantees the structure is valid, not that the content is correct.

Finally, notice how the error handling separates ValidationError from APIError. These are fundamentally different problems: a validation error means the model returned the wrong data shape (retry with a better prompt), while an API error means the network or rate limit failed (retry with backoff). Catching them separately lets you respond appropriately to each. Never catch a bare Exception — you'll mask bugs in your own code.

# pip install openai pydantic
from openai import OpenAI
import json
from pydantic import BaseModel, ValidationError
from typing import Optional

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

# Define the schema as both Pydantic model and tool definition
class ContactInfo(BaseModel):
    name: str
    email: str
    phone: Optional[str] = None
    company: Optional[str] = None
    role: Optional[str] = None

# Tool definition matches the Pydantic schema
extract_contact_tool = {
    "type": "function",
    "function": {
        "name": "extract_contact",
        "description": "Extract structured contact information from text.",
        "parameters": {
            "type": "object",
            "properties": {
                "name": {"type": "string", "description": "Full name of the person"},
                "email": {"type": "string", "description": "Email address"},
                "phone": {"type": "string", "description": "Phone number, if mentioned"},
                "company": {"type": "string", "description": "Company name, if mentioned"},
                "role": {"type": "string", "description": "Job title or role, if mentioned"},
            },
            "required": ["name", "email"],
        },
    },
}

def extract_contact(text: str) -> ContactInfo:
    """Extract contact info using tool use + Pydantic validation."""
    try:
        response = client.chat.completions.create(
            model="mistral",
            tools=[extract_contact_tool],
            tool_choice={"type": "function", "function": {"name": "extract_contact"}},
            messages=[{
                "role": "user",
                "content": f"Extract the contact information from this text:\n\n{text}"
            }]
        )

        # the structured args come back in message.tool_calls
        tool_calls = response.choices[0].message.tool_calls
        if tool_calls:
            args = json.loads(tool_calls[0].function.arguments)
            # Validate with Pydantic
            return ContactInfo(**args)

        raise ValueError("No tool call in response")

    except ValidationError as e:
        print(f"Validation failed: {e}")
        raise
    except Exception as e:
        print(f"API error: {e}")
        raise

# Usage
text = """
Best regards,
Jane Smith, VP of Engineering
TechCorp Inc. | jane.smith@techcorp.io | (555) 123-4567
"""

contact = extract_contact(text)
print(f"Name:    {contact.name}")
print(f"Email:   {contact.email}")
print(f"Phone:   {contact.phone}")
print(f"Company: {contact.company}")
print(f"Role:    {contact.role}")

// npm install openai zod
import OpenAI from 'openai';
import { z } from 'zod';

const client = new OpenAI({ baseURL: 'http://localhost:11434/v1', apiKey: 'ollama' });

// Define the schema with Zod
const ContactInfo = z.object({
  name: z.string(),
  email: z.string().email(),
  phone: z.string().optional(),
  company: z.string().optional(),
  role: z.string().optional(),
});

const extractContactTool = {
  type: 'function',
  function: {
    name: 'extract_contact',
    description: 'Extract structured contact information from text.',
    parameters: {
      type: 'object',
      properties: {
        name: { type: 'string', description: 'Full name of the person' },
        email: { type: 'string', description: 'Email address' },
        phone: { type: 'string', description: 'Phone number, if mentioned' },
        company: { type: 'string', description: 'Company name, if mentioned' },
        role: { type: 'string', description: 'Job title or role, if mentioned' },
      },
      required: ['name', 'email'],
    },
  },
};

async function extractContact(text) {
  try {
    const response = await client.chat.completions.create({
      model: 'mistral',
      tools: [extractContactTool],
      tool_choice: { type: 'function', function: { name: 'extract_contact' } },
      messages: [{
        role: 'user',
        content: `Extract the contact information from this text:\n\n${text}`
      }]
    });

    const toolCalls = response.choices[0].message.tool_calls;
    if (toolCalls) {
      // Validate with Zod
      const contact = ContactInfo.parse(JSON.parse(toolCalls[0].function.arguments));
      return contact;
    }
    throw new Error('No tool call in response');
  } catch (error) {
    if (error instanceof z.ZodError) {
      console.error('Validation failed:', error.issues);
    } else if (error?.status) {
      console.error(`API error: ${error.status} - ${error.message}`);
    }
    throw error;
  }
}

const text = `Best regards,
Jane Smith, VP of Engineering
TechCorp Inc. | jane.smith@techcorp.io | (555) 123-4567`;

const contact = await extractContact(text);
console.log(`Name:    ${contact.name}`);
console.log(`Email:   ${contact.email}`);
console.log(`Phone:   ${contact.phone}`);
console.log(`Company: ${contact.company}`);
console.log(`Role:    ${contact.role}`);

Adding Retry with Error Feedback

Now for the part that makes this production-ready: automatic error recovery. The code below wraps the extraction in a retry loop that runs up to max_retries times. Here's the clever part: on each retry, it appends the specific validation error to the prompt. So instead of blindly asking the model to try again, you're saying "the email field was null, but it's required — please fix that." the model reads the error and self-corrects. This approach resolves ~90% of failures on the first retry.

The other key detail is the time.sleep(2 ** attempt) after each failure. This is exponential backoff: 2 seconds, then 4, then 8. Without it, rapid retries during an API outage just make the problem worse — you'd trigger rate limiting on top of the original issue. And always set a max retry count. Without one, a consistently failing input (imagine someone passes in gibberish text) will loop forever, burning tokens and money with no hope of success.

from openai import OpenAI
import time, json
from pydantic import BaseModel, ValidationError
from typing import Optional

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

class ContactInfo(BaseModel):
    name: str
    email: str
    phone: Optional[str] = None
    company: Optional[str] = None
    role: Optional[str] = None

def extract_with_retry(text: str, max_retries: int = 3) -> ContactInfo:
    """Extract contact info with retry on validation failure."""
    last_error = None

    for attempt in range(1, max_retries + 1):
        prompt = f"Extract contact information from this text:\n\n{text}"

        # On retry, include the previous error
        if last_error:
            prompt += f"\n\nPrevious attempt failed with: {last_error}"
            prompt += "\nPlease fix the output to match the required schema exactly."

        try:
            response = client.chat.completions.create(
                model="mistral",
                tools=[{
                    "type": "function",
                    "function": {
                        "name": "extract_contact",
                        "description": "Extract contact info. ALL fields must be valid.",
                        "parameters": ContactInfo.model_json_schema(),
                    },
                }],
                tool_choice={"type": "function", "function": {"name": "extract_contact"}},
                messages=[{"role": "user", "content": prompt}],
            )

            tool_calls = response.choices[0].message.tool_calls
            if tool_calls:
                args = json.loads(tool_calls[0].function.arguments)
                contact = ContactInfo(**args)
                print(f"Attempt {attempt}: Success!")
                return contact

        except ValidationError as e:
            last_error = str(e)
            print(f"Attempt {attempt}: Validation error - {last_error}")
            time.sleep(2 ** attempt)  # Exponential backoff

        except Exception as e:
            print(f"Attempt {attempt}: API error - {e}")
            time.sleep(2 ** attempt)

    raise RuntimeError(f"Failed after {max_retries} attempts. Last error: {last_error}")

import OpenAI from 'openai';
import { z } from 'zod';

const client = new OpenAI({ baseURL: 'http://localhost:11434/v1', apiKey: 'ollama' });

const ContactInfo = z.object({
  name: z.string(),
  email: z.string().email(),
  phone: z.string().optional(),
  company: z.string().optional(),
  role: z.string().optional(),
});

async function extractWithRetry(text, maxRetries = 3) {
  let lastError = null;

  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    let prompt = `Extract contact information from this text:\n\n${text}`;

    if (lastError) {
      prompt += `\n\nPrevious attempt failed with: ${lastError}`;
      prompt += '\nPlease fix the output to match the required schema exactly.';
    }

    try {
      const response = await client.chat.completions.create({
        model: 'mistral',
        tools: [{
          type: 'function',
          function: {
            name: 'extract_contact',
            description: 'Extract contact info. ALL fields must be valid.',
            parameters: {
              type: 'object',
              properties: {
                name: { type: 'string' },
                email: { type: 'string' },
                phone: { type: 'string' },
                company: { type: 'string' },
                role: { type: 'string' },
              },
              required: ['name', 'email'],
            },
          },
        }],
        tool_choice: { type: 'function', function: { name: 'extract_contact' } },
        messages: [{ role: 'user', content: prompt }],
      });

      const toolCalls = response.choices[0].message.tool_calls;
      if (toolCalls) {
        const contact = ContactInfo.parse(JSON.parse(toolCalls[0].function.arguments));
        console.log(`Attempt ${attempt}: Success!`);
        return contact;
      }
    } catch (error) {
      if (error instanceof z.ZodError) {
        lastError = error.issues.map(i => `${i.path}: ${i.message}`).join(', ');
        console.log(`Attempt ${attempt}: Validation error - ${lastError}`);
      } else if (error?.status) {
        console.log(`Attempt ${attempt}: API error - ${error.message}`);
      } else { throw error; }
      await new Promise(r => setTimeout(r, 2 ** attempt * 1000));
    }
  }
  throw new Error(`Failed after ${maxRetries} attempts. Last error: ${lastError}`);
}

Hands-On Exercise

Environment Setup

# Python
pip install openai "pydantic>=2.0"
# No API key needed — Ollama runs locally (ollama serve)

# Node.js
npm install openai zod
# No API key needed — Ollama runs locally (ollama serve)

Step 1: Define the Schema and Test Data

Before extracting anything, you need a data model and test cases. This step defines the ContactInfo schema and 5 real-world email signatures that range from easy to tricky. Having a fixed test set lets you objectively compare prompt-only vs tool-use approaches in Step 2.

Create a new file called extractor.py (or extractor.mjs):

from openai import OpenAI
import json
from pydantic import BaseModel, ValidationError
from typing import Optional

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

class ContactInfo(BaseModel):
    name: str
    email: str
    phone: Optional[str] = None
    company: Optional[str] = None
    role: Optional[str] = None

# 5 test email signatures — easy to hard
TEST_SIGNATURES = [
    "Best, Jane Smith | jane@acme.com | Acme Corp",
    "John Doe, Senior Engineer at MegaTech\njohn.doe@megatech.io | (555) 234-5678",
    "Cheers,\nDr. Maria García-López, Head of Research\nBioGen International\nmgarcia@biogen.int",
    "— Alex K. | Product @ StartupXYZ | alex@startupxyz.co | they/them",
    "Thanks!\nRobert \"Bob\" Williams III\nChief Financial Officer\nGlobal Finance Partners LLC\nrwilliams@gfp.com\n+1 (212) 555-0199",
]

print(f"Schema: {json.dumps(ContactInfo.model_json_schema(), indent=2)}")
print(f"\nTest signatures: {len(TEST_SIGNATURES)}")

import OpenAI from 'openai';
import { z } from 'zod';

const client = new OpenAI({ baseURL: 'http://localhost:11434/v1', apiKey: 'ollama' });

const ContactInfo = z.object({
  name: z.string(),
  email: z.string().email(),
  phone: z.string().optional(),
  company: z.string().optional(),
  role: z.string().optional(),
});

const TEST_SIGNATURES = [
  "Best, Jane Smith | jane@acme.com | Acme Corp",
  "John Doe, Senior Engineer at MegaTech\njohn.doe@megatech.io | (555) 234-5678",
  "Cheers,\nDr. Maria García-López, Head of Research\nBioGen International\nmgarcia@biogen.int",
  "— Alex K. | Product @ StartupXYZ | alex@startupxyz.co | they/them",
  "Thanks!\nRobert \"Bob\" Williams III\nChief Financial Officer\nGlobal Finance Partners LLC\nrwilliams@gfp.com\n+1 (212) 555-0199",
];

console.log(`Test signatures: ${TEST_SIGNATURES.length}`);

Run it: python extractor.py (or node extractor.mjs)

Step 2: Extract with Tool Use + Validation

Now let's build the extraction function and see how it handles the full range of signatures. The interesting question isn't whether it works on the clean ones (Sig 1 is trivial) — it's whether it can handle Dr. María García-López's hyphenated name and Robert "Bob" Williams III's nickname in quotes. These are the cases where regex-based extraction falls apart, and tool use shines.

This function combines the tool definition, forced tool_choice, and Pydantic validation from the code walkthrough into a single reusable function. It uses the ContactInfo model and TEST_SIGNATURES from Step 1.

Add the following to extractor.py:

extract_tool = {
    "type": "function",
    "function": {
        "name": "extract_contact",
        "description": "Extract structured contact information from an email signature.",
        "parameters": ContactInfo.model_json_schema(),
    },
}

def extract_contact(text: str) -> ContactInfo:
    """Extract contact info using forced tool use + Pydantic validation."""
    response = client.chat.completions.create(
        model="mistral",
        tools=[extract_tool],
        tool_choice={"type": "function", "function": {"name": "extract_contact"}},
        messages=[{"role": "user", "content": f"Extract contact info:\n\n{text}"}],
    )
    tool_calls = response.choices[0].message.tool_calls
    if tool_calls:
        return ContactInfo(**json.loads(tool_calls[0].function.arguments))
    raise ValueError("No tool call in response")

# Run against all 5 test signatures
successes = 0
for i, sig in enumerate(TEST_SIGNATURES, 1):
    try:
        contact = extract_contact(sig)
        print(f"✓ Sig {i}: {contact.name} <{contact.email}> @ {contact.company or 'N/A'}")
        successes += 1
    except (ValidationError, ValueError) as e:
        print(f"✗ Sig {i}: FAILED — {e}")
    except Exception as e:
        print(f"✗ Sig {i}: API error — {e}")

print(f"\nResults: {successes}/{len(TEST_SIGNATURES)} extracted successfully")

const extractTool = {
  type: 'function',
  function: {
    name: 'extract_contact',
    description: 'Extract structured contact information from an email signature.',
    parameters: {
      type: 'object',
      properties: {
        name: { type: 'string', description: 'Full name' },
        email: { type: 'string', description: 'Email address' },
        phone: { type: 'string', description: 'Phone number if present' },
        company: { type: 'string', description: 'Company name if present' },
        role: { type: 'string', description: 'Job title if present' },
      },
      required: ['name', 'email'],
    },
  },
};

async function extractContact(text) {
  const response = await client.chat.completions.create({
    model: 'mistral',
    tools: [extractTool],
    tool_choice: { type: 'function', function: { name: 'extract_contact' } },
    messages: [{ role: 'user', content: `Extract contact info:\n\n${text}` }],
  });
  const toolCalls = response.choices[0].message.tool_calls;
  if (toolCalls) return ContactInfo.parse(JSON.parse(toolCalls[0].function.arguments));
  throw new Error('No tool call');
}

let successes = 0;
for (let i = 0; i < TEST_SIGNATURES.length; i++) {
  try {
    const contact = await extractContact(TEST_SIGNATURES[i]);
    console.log(`✓ Sig ${i+1}: ${contact.name} <${contact.email}> @ ${contact.company || 'N/A'}`);
    successes++;
  } catch (error) {
    console.log(`✗ Sig ${i+1}: FAILED — ${error.message?.slice(0, 80)}`);
  }
}
console.log(`\nResults: ${successes}/${TEST_SIGNATURES.length} extracted successfully`);

Run it: python extractor.py (or node extractor.mjs)

Step 3: Add Retry with Error Feedback

Even with tool use, validation can occasionally fail. Maybe the model returns an empty string for a required field, or interprets "j (at) co (dot) com" too literally. This step wraps the extraction in a retry loop that feeds specific error messages back to the model. Here's the key idea: instead of blindly retrying, we tell the model exactly what went wrong so it can fix the specific problem. This step builds on extract_contact() from Step 2.

Add the following to extractor.py:

import time

def extract_with_retry(text: str, max_retries: int = 3) -> ContactInfo:
    """Extract with automatic retry on validation failure."""
    last_error = None
    for attempt in range(1, max_retries + 1):
        prompt = f"Extract contact info:\n\n{text}"
        if last_error:
            prompt += f"\n\nPrevious attempt failed: {last_error}\nFix the output."
        try:
            response = client.chat.completions.create(
                model="mistral",
                tools=[extract_tool],
                tool_choice={"type": "function", "function": {"name": "extract_contact"}},
                messages=[{"role": "user", "content": prompt}],
            )
            tool_calls = response.choices[0].message.tool_calls
            if tool_calls:
                return ContactInfo(**json.loads(tool_calls[0].function.arguments))
        except ValidationError as e:
            last_error = str(e)
            print(f"  Attempt {attempt}: Validation error, retrying...")
            time.sleep(2 ** attempt)
        except Exception as e:
            print(f"  Attempt {attempt}: API error — {e}")
            time.sleep(2 ** attempt)
    raise RuntimeError(f"Failed after {max_retries} attempts: {last_error}")

# Test retry with a deliberately tricky signature
tricky = "Contact: J. at some-company, email is j (at) co (dot) com, phone TBD"
try:
    result = extract_with_retry(tricky)
    print(f"Extracted: {result.name} <{result.email}>")
except RuntimeError as e:
    print(f"Gave up: {e}")

async function extractWithRetry(text, maxRetries = 3) {
  let lastError = null;
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    let prompt = `Extract contact info:\n\n${text}`;
    if (lastError) prompt += `\n\nPrevious attempt failed: ${lastError}\nFix the output.`;
    try {
      const response = await client.chat.completions.create({
        model: 'mistral',
        tools: [extractTool],
        tool_choice: { type: 'function', function: { name: 'extract_contact' } },
        messages: [{ role: 'user', content: prompt }],
      });
      const toolCalls = response.choices[0].message.tool_calls;
      if (toolCalls) return ContactInfo.parse(JSON.parse(toolCalls[0].function.arguments));
    } catch (error) {
      if (error instanceof z.ZodError) {
        lastError = error.issues.map(i => `${i.path}: ${i.message}`).join(', ');
        console.log(`  Attempt ${attempt}: Validation error, retrying...`);
      } else { throw error; }
      await new Promise(r => setTimeout(r, 2 ** attempt * 1000));
    }
  }
  throw new Error(`Failed after ${maxRetries} attempts: ${lastError}`);
}

const tricky = "Contact: J. at some-company, email is j (at) co (dot) com, phone TBD";
try {
  const result = await extractWithRetry(tricky);
  console.log(`Extracted: ${result.name} <${result.email}>`);
} catch (error) {
  console.log(`Gave up: ${error.message}`);
}

Run it: python extractor.py (or node extractor.mjs)

Verify Everything Works

Run the complete file to execute all steps in sequence:

python extractor.py    # or: node extractor.mjs

Stretch Goals (Optional)

• Build a generic extract(text, schema) function that accepts any Pydantic/Zod schema
• Add a confidence score to each extracted field based on the source text

Multi-Modal Input: Vision and PDF

Agents don't only consume text. the model can process imagesBinary visual data (JPEG, PNG, GIF, WebP) sent to the model as base64-encoded strings or URLs, enabling the model to "see" and reason about visual content. and PDFsPortable Document Format files that the model can read natively, extracting text, tables, and layout information without external OCR. directly, which unlocks powerful agent use cases like reading scanned documents, extracting data from photographs, and analyzing charts or diagrams.

Sending Images via Base64

To send an image to the model, you encode it as a base64 string and include it in the message's content array with type: "image" and the appropriate media_type (e.g., image/jpeg, image/png). This lets agents process photos of receipts, screenshots of dashboards, scanned filings, or any visual input without needing a separate OCR pipeline.

Vision Use Cases for Agents

• Scanned document extraction — read a scanned UCC filing or invoice and extract structured fields like filing numbers, debtor names, and dates
• Photo-based data capture — extract product details from a photo of a shipping label or whiteboard notes from a meeting
• Chart and diagram analysis — interpret a bar chart image and return the underlying data as JSON

PDF Processing

The model supports native document understanding for PDFs, so you can send multi-page documents directly without external OCR tools. This is especially valuable for agents that need to process contracts, medical records, or government filings. Combine PDF input with tool use to extract structured data automatically.

# Pseudocode: an analyze_document tool for a multi-modal agent

TOOL analyze_document(document_base64, media_type):
    """Send a scanned document to the model with vision and
    return structured data extracted from it."""

    SEND message to the model:
        content = [
            { type: "image", source: base64_data, media_type: media_type },
            { type: "text", text: "Extract: filing number, debtor name,
              secured party, filing date. Return as JSON." }
        ]
        tools = [ structured_filing_schema ]

    PARSE tool call response
    VALIDATE against schema
    RETURN structured filing record

Knowledge Check

Test your understanding of structured output, validation, and error recovery.

class Product(BaseModel):
    name: str
    price: float
    in_stock: bool

response = client.chat.completions.create(
    model="mistral",
    tools=[my_tool],
    ______={"type": "function", "function": {"name": "my_tool"}},
    messages=[...]
)

Module Summary