M07 — Production

Acme Support has tools, structured output, memory, retrieval, and a team of specialists — but it's still demo code. Production adds four things a demo skips: streaming so it feels instant, reliability so a hiccup doesn't crash it, cost control so it's affordable, and a real path to ship. This is the last module — time to make Acme deployable.

Learning Objectives

  • Stream responses token-by-token in all three SDKs, and explain why it transforms perceived latency
  • Handle failures with each SDK's typed exceptions, and configure the built-in retry behavior
  • Read token usage per provider and name the levers that cut cost (caching, model tiers, trimming history)
  • Know the path from script to shipped service: the agent frameworks, deployment shapes, and observability
  • Walk a concrete production checklist before putting an agent in front of real customers

From Demo to Production

Everyday Analogy

BEFORE: A concept car on a show-floor turntable looks finished. It gleams, the doors open, the screens light up.

PAIN: But it has no airbags, no crash testing, no fuel gauge, and it's never driven in the rain. It's a demo of an idea, not a car you'd put your family in. The gap between "looks done" and "safe to ship" is exactly the unglamorous stuff: safety systems, gauges, failure handling, a factory line.

MAPPING: Your agent from M00–M06 is the concept car. It works in the demo. Production is the airbags and fuel gauge: it streams so the user isn't staring at a blank screen (responsiveness), it survives a rate-limit or network blip (reliability), it tells you what it costs (the gauge), and it runs somewhere real (the factory line). None of it is new agent theory — it's the engineering that makes the theory survive contact with real users.

Streaming — Responsiveness

By default a model call blocks until the entire answer is ready, then returns it. For a long reply that's several seconds of blank screen. Streaming delivers the answer token-by-token as it's generated, so the user sees words within a fraction of a second. The total time is the same — but the perceived latencyHow fast it *feels*. Time-to-first-token matters far more to users than total time. Streaming slashes time-to-first-token even though total generation time is unchanged. collapses.

Same answer, same total time — but watch how differently the two feel. The blocking version shows nothing until the end; the streaming version fills in from the first moment:

Blocking vs Streaming — Time to First Token
⏳ Blocking — wait, then everything at once
generating… Your refund for AC-1042 is on its way.
time to first token: ~3.0s
⚡ Streaming — words appear as they're generated
Your refund for AC-1042 is on its way.
time to first token: ~0.3s

Here's streaming in all three SDKs. The shape is the same everywhere: start a stream, iterate the pieces as they arrive, print each immediately.

# Anthropic — messages.stream() context manager; iterate text_stream.
from anthropic import Anthropic
client = Anthropic()

with client.messages.stream(
    model="claude-sonnet-5", max_tokens=512,
    messages=[{"role": "user", "content": "Summarize our return policy."}],
) as stream:
    for text in stream.text_stream:          # each token as it arrives
        print(text, end="", flush=True)
    final = stream.get_final_message()        # full message once done (usage, etc.)
// Anthropic — messages.stream(); the "text" event gives you each delta.
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();

const stream = client.messages.stream({
  model: "claude-sonnet-5", max_tokens: 512,
  messages: [{ role: "user", content: "Summarize our return policy." }],
});
stream.on("text", (delta) => process.stdout.write(delta));  // each token
const final = await stream.finalMessage();                  // full message once done
# Gemini — generate_content_stream(); iterate chunks, read chunk.text.
from google import genai
client = genai.Client()

for chunk in client.models.generate_content_stream(
    model="gemini-2.5-flash",
    contents="Summarize our return policy.",
):
    print(chunk.text, end="", flush=True)     # each chunk as it arrives
# usage_metadata is populated on the final chunk
// Gemini — generateContentStream() returns an async iterable directly.
import { GoogleGenAI } from "@google/genai";
const ai = new GoogleGenAI({});

const stream = await ai.models.generateContentStream({
  model: "gemini-2.5-flash",
  contents: "Summarize our return policy.",
});
for await (const chunk of stream) {
  process.stdout.write(chunk.text ?? "");     // each chunk as it arrives
}
# OpenAI (Responses) — stream=True yields typed events; filter text deltas.
from openai import OpenAI
client = OpenAI()

stream = client.responses.create(
    model="gpt-5.5", input="Summarize our return policy.", stream=True,
)
for event in stream:
    if event.type == "response.output_text.delta":  # the text-delta event
        print(event.delta, end="", flush=True)
// OpenAI (Responses) — stream: true; filter the text-delta event type.
import OpenAI from "openai";
const client = new OpenAI();

const stream = await client.responses.create({
  model: "gpt-5.5", input: "Summarize our return policy.", stream: true,
});
for await (const event of stream) {
  if (event.type === "response.output_text.delta") {
    process.stdout.write(event.delta);        // each text delta
  }
}
What Just Happened?

Anthropic hands you a clean text_stream / "text" event; Gemini gives you chunks with .text; OpenAI emits typed events and you filter for response.output_text.delta. Same idea, three surfaces. For a chat UI this is non-negotiable — a 4-second blank box reads as "broken," while text appearing in 300ms reads as "thinking out loud." Streaming is also how you dodge SDK timeouts on very long outputs (a reason the docs push it for big max_tokens).

Reliability — Errors & Retries

Networks blip. Rate limits trip. Servers hiccup. A demo ignores this; production expects it. Good news: all three SDKs auto-retry transient failures (429s and 5xx) with exponential backoff, and expose typed exceptions so you can react precisely. You mostly configure the retries and catch the exceptions you care about.

import anthropic
client = anthropic.Anthropic(max_retries=3)      # auto-retries 429/5xx (default 2)

try:
    resp = client.messages.create(model="claude-sonnet-5", max_tokens=512,
                                  messages=[{"role": "user", "content": "Hi"}])
except anthropic.RateLimitError as e:
    print("rate limited:", e.status_code)         # back off / queue the request
except anthropic.APIStatusError as e:
    print("API error:", e.status_code)            # any other non-2xx
except anthropic.APIConnectionError:
    print("network problem — SDK already retried and gave up")
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({ maxRetries: 3 });  // auto-retries 429/5xx (default 2)

try {
  await client.messages.create({ model: "claude-sonnet-5", max_tokens: 512,
    messages: [{ role: "user", content: "Hi" }] });
} catch (err) {
  if (err instanceof Anthropic.RateLimitError) console.log("rate limited:", err.status);
  else if (err instanceof Anthropic.APIError)  console.log("API error:", err.status);
}
# Gemini — exceptions live in google.genai.errors. NOTE: field is .code, not .status_code.
from google import genai
from google.genai import errors, types

client = genai.Client(http_options=types.HttpOptions(   # set retries explicitly (see note)
    retry_options=types.HttpRetryOptions(attempts=5)))

try:
    resp = client.models.generate_content(model="gemini-2.5-flash", contents="Hi")
except errors.ClientError as e:        # 4xx (e.g. 429)
    print("client error:", e.code, e.message)
except errors.ServerError as e:        # 5xx
    print("server error:", e.code)
// Gemini — ApiError from @google/genai; status on .status.
import { GoogleGenAI, ApiError } from "@google/genai";

const ai = new GoogleGenAI({
  httpOptions: { retryOptions: { attempts: 5 } },   // set retries explicitly
});

try {
  await ai.models.generateContent({ model: "gemini-2.5-flash", contents: "Hi" });
} catch (err) {
  if (err instanceof ApiError) console.log("API error:", err.status, err.message);
}
import openai
client = openai.OpenAI(max_retries=3)             # auto-retries 429/5xx (default 2)

try:
    resp = client.responses.create(model="gpt-5.5", input="Hi")
except openai.RateLimitError as e:
    print("rate limited:", e.status_code)
except openai.APIStatusError as e:
    print("API error:", e.status_code)
except openai.APIConnectionError as e:
    print("network:", e.__cause__)                # SDK already retried
import OpenAI from "openai";
const client = new OpenAI({ maxRetries: 3 });     // auto-retries 429/5xx (default 2)

try {
  await client.responses.create({ model: "gpt-5.5", input: "Hi" });
} catch (err) {
  if (err instanceof OpenAI.RateLimitError) console.log("rate limited:", err.status);
  else if (err instanceof OpenAI.APIError)  console.log("API error:", err.status);
}
Two Small Traps
  • Gemini reads status as .code (Python) — not .status_code like Anthropic/OpenAI (whose JS SDKs use .status). Small difference, real bug if you copy-paste across providers.
  • Gemini's default auto-retry is version-dependent. Anthropic and OpenAI default to max_retries=2; for Gemini, set retry_options / retryOptions explicitly so behavior is deterministic across SDK versions.
Two More Layers You Already Learned

API errors are only one layer. Remember: tool errors should be returned as data so the agent recovers (M01), and the agent loop needs a cap so it can't spin forever (M03). Production reliability = SDK retries plus your loop cap plus graceful tool failures plus a "hand off to a human" fallback when all else fails.

Cost

Every token in and out is billed. In production you watch it, and you have levers to lower it. First, read what each call cost — the usage object is on every response:

 AnthropicGoogle GeminiOpenAI
Usage objectresp.usageresp.usage_metadataresp.usage
Input tokens.input_tokens.prompt_token_count.input_tokens
Output tokens.output_tokens.candidates_token_count.output_tokens
Totalsum them.total_token_count.total_tokens
Four Levers That Cut the Bill
  1. Pick the right model tier. A support triage step doesn't need the flagship. Use a cheaper/faster tier (Haiku, Flash-lite, mini/nano) for routing and simple turns; save the big model for hard ones. Multi-agent (M06) lets you assign a different tier per specialist.
  2. Prompt caching. If a big system prompt or knowledge chunk repeats across calls, cache it — cached input tokens cost a fraction of fresh ones (Anthropic's cache_control; the others have their own caching). Huge for RAG and long system prompts.
  3. Trim the history. Remember M04: resending the whole transcript every turn is billed every turn. Cap or summarize (compaction) old turns so a long chat doesn't balloon input cost.
  4. Retrieve narrowly. RAG (M05) with tight top-k sends only the passages you need, not the whole doc — fewer input tokens and a sharper answer.

Ship It

You have a hardened agent. Where does it run? Three concerns close the loop from script to service.

From Script to Service
  • Use a framework for the real build. The hand-rolled loops taught you what's happening; for production, the agent frameworks from M06 (Claude Agent SDK, OpenAI Agents SDK, Google ADK) give you handoffs, sessions, and built-in tracing for free. You understand the machinery now, so a framework is leverage, not a black box.
  • Deploy as a service. Wrap your agent in a small web API (FastAPI, Express) behind an endpoint, and run it on serverless (Cloud Run, Lambda) or a container. Stream responses over Server-Sent Events or WebSockets to the browser. Keep API keys in a secrets manager, never in the image.
  • Observe it. Log every request id, token count, latency, tool call, and error. Watch cost and error-rate dashboards. When something goes wrong at 2am, traces of the tool loop are what save you — which is exactly why the frameworks build tracing in.
Before Real Customers — The Non-Negotiables

Two things this course flagged but production makes mandatory: gate write-tools behind confirmation (M03's process_refund should not fire from a persuasive message alone — require human approval or hard validation), and ground factual answers (M05's RAG, with an "only from context" instruction, so the bot says "let me check" instead of inventing a policy). A confident, wrong, money-moving agent is worse than no agent.

Production Checklist

Run down this list before Acme talks to a real customer. Every item traces back to a module you've completed:

Ship-Ready Acme Support

  • Streaming on for user-facing replies (M07) — no blank-screen waits
  • Typed error handling + retries configured; loop cap + human-handoff fallback (M07, M03)
  • Structured output where a UI/DB consumes the answer (M02)
  • Write-tools gated behind confirmation/validation (M03) — refunds don't auto-fire
  • Grounded in your docs via RAG with "only from context" (M05) — no invented policies
  • History bounded (cap or compaction) so cost doesn't balloon (M04)
  • Cost watched: usage logged, right model tier per step, caching on repeated context (M07)
  • Observability: request ids, latency, tool traces, error rate (M07)
  • Secrets in a manager, never in code or images (M00)

Three Ways, One Idea

ConceptAnthropicGoogle GeminiOpenAI
Stream methodmessages.stream()generate_content_stream()responses.create(stream=True)
Read a deltatext_stream / "text" eventchunk.textevent response.output_text.delta
Error base classAPIError / APIStatusErrorAPIError (ClientError/ServerError)APIError / APIStatusError
Read status.status_code / JS .status.code / JS .status.status_code / JS .status
Default retries2set explicitly2
Usage objectusageusage_metadatausage
Agent frameworkClaude Agent SDKADKAgents SDK
Why the Differences?

By now the pattern is familiar: the production surfaces rhyme across all three — every SDK streams, auto-retries, exposes typed errors, and reports usage — while the exact nouns (text_stream vs chunk.text vs a typed event; .status_code vs .code; usage vs usage_metadata) differ just enough to trip a copy-paste. That's been the whole course in miniature: learn the production concern once, translate the field names per provider. You now do that fluently.

Knowledge Check

Q1: Why does streaming matter, given the total generation time is the same?

AIt makes the model generate more tokens
BIt collapses perceived latency — time-to-first-token drops from seconds to ~300ms, so it feels instant
CIt's cheaper per token
DIt removes the need for error handling
Correct! Users judge responsiveness by time-to-first-token, not total time. Streaming shows words immediately, turning a "broken"-feeling blank screen into a "thinking out loud" experience.

Q2: A 429 (rate limit) hits mid-request. What do the SDKs do by default?

ACrash immediately
BReturn an empty response silently
CAuto-retry with exponential backoff (Anthropic/OpenAI default 2; set Gemini explicitly)
DSwitch providers automatically
Correct! All three auto-retry transient 429/5xx errors with backoff. Anthropic and OpenAI default to 2 retries; for Gemini, configure retry_options explicitly since the default is version-dependent.

Q3: Which is the odd-one-out field name when reading an error's HTTP status in Python?

AAnthropic .status_code
BOpenAI .status_code
CThey're all .status_code
DGemini uses .code, not .status_code
Correct! Gemini's Python errors expose .code; Anthropic and OpenAI use .status_code (their JS SDKs use .status). A classic copy-paste bug across providers.

Q4: Which lever does NOT reduce token cost?

ATurning on streaming
BUsing a cheaper model tier for simple steps
CPrompt caching repeated context
DTrimming or compacting long history
Correct! Streaming changes when tokens arrive, not how many — same token count, same cost. The real levers are model tier, caching, trimming history, and narrow retrieval.

Q5: Which two safeguards does the module call "non-negotiable" before real customers?

AStreaming and caching
BGate write-tools behind confirmation, and ground factual answers in your docs (RAG)
CUse the flagship model and disable retries
DMulti-agent and structured output
Correct! A confident, wrong, money-moving agent is worse than none. Require approval for write-tools, and ground factual answers so it says "let me check" instead of inventing policy.

You Finished the Course 🎉

One Agent, Three SDKs — Complete

You started at "install three SDKs and say hello." You're ending with a support agent that uses tools, returns typed data, orchestrates a toolbox, remembers conversations, grounds answers in a knowledge base, routes to specialists, and is hardened for production — and you can build every piece of it in Anthropic, Google Gemini, and OpenAI, translating fluently between them.

The arc you walked:

  • M00–M01: three clients, and the tool-use loop that is an agent
  • M02–M03: typed output, and orchestrating many tools
  • M04–M05: memory across turns, and grounding in your own docs
  • M06–M07: teams of specialist agents, and shipping to production

The one idea to keep: every provider implements the same agent pipeline — send, maybe call a tool, loop, answer. The branded nouns (tool_use vs function_call, messages vs contents vs input, usage vs usage_metadata) are translation, not re-learning. That fluency is your real takeaway: you're not locked to one vendor.

Where to go next: pick the provider your project needs (or mix them — Voyage embeddings + Claude, anyone?), build a real agent for your own domain, and lean on each SDK's agent framework once you know what it's doing under the hood. You do now. Go ship something.