The 6 concepts of shipping an agent

Tap any concept to jump to its 5-card cluster.

• 1API ProtocolsREST, SSE, and WebSocket — pick the right channel
• 2ContainerizationDocker as the shipping container of software
• 3Cloud DeploymentLambda vs Cloud Run vs Railway / VM
• 4Scaling & QueuesConcurrent requests, queues, rate limits
• 5Streaming ResponsesToken-by-token delivery via SSE
• 6Auth & AuthorizationAuthenticate, authorize, throttle

How clients talk to your agent

An agent reply takes 5–45 seconds. The wire protocol you pick decides whether the user stares at a blank screen or watches tokens stream in. Three options exist: REST (one request, one full response), SSE (server pushes events over a long-lived HTTP connection), and WebSocket (full duplex; both sides can send anytime).

For 95% of agent traffic the right answer is SSE. It rides on plain HTTP, traverses every proxy and CDN, and matches the upstream Anthropic streaming API one-for-one. REST is for stateless side calls (health, feedback). WebSocket only earns its complexity when the client must send signals during the stream.

Letter vs. radio vs. phone call

BEFORE: You need updates from a friend across town. You could mail a letter, ask "anything new?" and wait days for the reply. That is REST polling — clean, simple, but every poll costs a stamp even when the answer is "nothing yet."

PAIN: If your friend is writing a long story, sending a new letter every five seconds wastes paper and clogs the mailbox. A live phone call (WebSocket) fixes the wait but ties up the line continuously, even during silence.

MAPPING: The sweet spot is a radio broadcast (SSE). Your friend speaks into the mic, you listen on a one-way channel, you hear each sentence as it lands, and the channel closes cleanly when the message ends. No polling waste, no idle phone line. That matches how an agent actually generates — one direction, in chunks, until done.

What an SSE stream looks like

1. Client opens one HTTP requestPOST or GET with header Accept: text/event-stream. The connection stays open instead of closing after the response.
2. Server responds with event-stream content typeContent-Type: text/event-stream. Now the channel is "open mic" until the server closes it.
3. Server writes named events as they happenEach event has an optional event: name (token, tool_call, done) and a data: JSON payload. Lines flush immediately.
4. Browser parses with EventSourceThe native EventSource API auto-reconnects on drop and dispatches each event to a JavaScript handler.

REST vs SSE vs WebSocket

# Question: which protocol for this endpoint?

IF endpoint is stateless side call:
  # /health, /feedback, /history
  USE REST  # request → full response → close

ELIF server pushes tokens, client only listens:
  # /chat with token-by-token output
  USE SSE   # default for 95% of agent calls

ELIF client must send mid-stream signals:
  # cancel, mid-run param tweak, live tool result
  USE WebSocket  # pay the complexity tax

ELSE:
  DEFAULT to SSE  # simpler than WS, streams better than REST

Closing the SSE connection is itself a "stop generating" signal — you almost never need WebSocket just for cancel.

Misconceptions

Pack the agent into a portable box

Your laptop has Python 3.12 and libssl 3.0. The production VM has Python 3.10 and libssl 1.1. Without a container, "works on my machine" is a feature, not a bug. Docker fixes this by packaging your agent code, runtime, and every shared library into one immutable image — a frozen snapshot that runs identically on a Mac, in CI, and on any cloud.

Each image is built from a small declarative script (the Dockerfile). Each instruction adds a cached layer; if only your code changed, Docker reuses the dependency layer from yesterday and rebuilds in seconds. A running copy of the image is a container — an isolated process that shares the host kernel (so it boots in milliseconds, unlike a VM).

The shipping container revolution

BEFORE: Before standardized shipping containers in the 1950s, loading a cargo ship was chaos. Every crate had a different shape, fragile goods needed special crews, and dockworkers spent days hand-packing irregular cargo into ship holds.

PAIN: Moving cargo between a ship and a train meant unpacking and repacking the entire load. Software has the identical disease: your agent depends on a precise mix of language runtime, OS libraries, and config that exists only on your machine. Re-creating that on a server is the modern version of hand-packing crates.

MAPPING: Docker is the shipping container of software. You pack the agent, its Python interpreter, its dependencies, and its config into one standardized box. That box drops onto Cloud Run, Lambda, ECS, your laptop, or a coworker's CI runner without a single repack — exactly like a sealed shipping container fits any truck, ship, or train.

Source → image → running container

1. Write a DockerfileA text file listing every step: pick a base image, copy dependency manifest, install, copy code, set the start command.
2. Build the imageEach instruction becomes a cached layer. If only your code changed, the dependency layer is reused. 3-minute build becomes a 10-second rebuild.
3. Push to a registryTag the image and push it to a registry (Artifact Registry, ECR, Docker Hub) so cloud platforms can pull it on deploy.
4. Run as a containerThe cloud platform pulls the image and starts one or more isolated processes from it. Same image, many parallel containers — that is how autoscale works.
5. Inject secrets at runtimeAPI keys live in env vars or a secrets manager — never inside the image. .dockerignore keeps .env files out of the build context.

An agent image, layer by layer

# WHAT: Pack an agent into a portable, layered image
# WHY:  Identical runtime on laptop, CI, and cloud

START FROM slim python base
  # 150 MB, not 900 MB; fewer CVEs to patch

COPY dependency manifest FIRST
RUN install dependencies
  # cached layer; rebuilds skip when code-only changes

COPY agent source code
CREATE non-root user, switch to it
  # exploit blast radius shrinks dramatically

DECLARE healthcheck endpoint
EXPOSE port 8080
SET start command RUN agent server

# GOTCHA: never bake secrets here. Even after RUN rm,
# the original layer still contains the file.

Misconceptions

Where the container runs

You have an image. Now you choose a host. Three categories cover almost every agent: serverless functions (Lambda) bill per invocation but cap timeouts at 15 minutes and limit streaming. Managed containers (Cloud Run, Azure Container Apps) auto-scale your image from zero to thousands, support 60-minute timeouts, and stream natively. VMs / Railway stay always-on with no cold start but cost the same whether traffic is 0 or 10,000 RPS.

For most agent APIs Cloud Run wins: streaming, scale-to-zero pricing, generous timeouts, one-line deploy. Lambda is right for short synchronous tasks. VMs make sense only for steady high-traffic workloads where the always-on cost is lower than serverless per-invocation pricing.

Renting, leasing, or buying a car

BEFORE: You need a vehicle for occasional trips. Renting by the hour (Lambda) means you only pay when driving — perfect for short, occasional drives. But every rental starts with paperwork (cold start) and you cannot keep the car past 15 minutes per trip.

PAIN: Most agent calls take 30–60 seconds of multi-step reasoning, sometimes much more. Lambda's 15-minute cap technically fits, but its limited streaming means tokens cannot reach the user the way they expect. Buying the car outright (a dedicated VM) lets you drive forever, but you pay even when it sits in the garage overnight.

MAPPING: Cloud Run is leasing. The car appears when you need it (auto-spin-up), can stay with you for up to 60 minutes per trip, and returns to the lot when you are done (scale-to-zero). It supports the open-window streaming agents need, and it costs nothing while idle. For 80% of agent products, this is the right vehicle.

From image to live URL

1. Push image to a registryTag your image and upload it to Artifact Registry, ECR, or Docker Hub. The cloud platform will pull from here.
2. Deploy with one commandCloud Run, App Runner, or Container Apps each take an image URL and a region, then return a live HTTPS URL with TLS terminated for you.
3. Cold start the first instanceFirst request after idle wakes a container from scratch — pulls the image, boots the runtime, runs your start command. Users feel 1–5 seconds.
4. Auto-scale on demandAs concurrent requests rise, the platform spins up more identical containers in parallel. Idle instances are killed within minutes; you stop paying.
5. Roll back instantlyEach deploy creates a new revision. Traffic split lets you canary 5% to v2, watch metrics, then flip 100% — or rollback to v1 in one command.

Lambda vs Cloud Run vs Railway / VM

Cloud Run is the sweet spot. For 10K agent calls/day at ~30s each, expect ~$15–30/month vs $50+/month for an always-on VM.

Misconceptions

Scale on concurrency, not CPU

Agent handlers spend 95% of their time waiting — for Claude's API, for tool calls, for database queries. CPU usage stays at 5–10% even when the instance is fully saturated with in-flight requests. If you scale on CPU like a normal web app, you will never add instances and your queue will quietly grow forever.

The right signal is concurrent requests per instance. Cloud Run's --concurrency=10 says: each instance handles at most 10 simultaneous requests; the 11th triggers a new instance. Add a queue in front to absorb bursts, plus exponential backoff and a circuit breaker on the upstream Claude API to handle 429s gracefully.

The restaurant with one kitchen

BEFORE: A restaurant. Waiters are your server instances — each can carry a few tables at once. Quiet Monday: one waiter is plenty. Busy Saturday: ten.

PAIN: But the kitchen (Anthropic API) has a fixed cook rate. If all ten waiters slam orders in at once, the kitchen sends back "we're overwhelmed" tickets — that is HTTP 429. Hiring more waiters does not help if the kitchen is the bottleneck.

MAPPING: The fix is a reservation system (a task queue). Orders enter the queue and leave at a rate the kitchen can match. When the kitchen flashes "slow down," the queue backs off automatically (exponential backoff). And a circuit breaker stops sending entirely if the kitchen has truly fallen over — better to pause than flood it with retries.

Queue + worker + backoff + breaker

1. API server enqueues the taskAccept the HTTP request, push a task object onto Redis / SQS / Cloud Tasks, return a job_id. The HTTP request closes in milliseconds.
2. Workers pull at a controlled rateA pool of worker processes pulls tasks one at a time and runs the agent. Worker count autoscales on queue depth.
3. Honor retry-after on 429If Claude returns 429 with a retry-after header, sleep that long. With no header, exponential backoff: 1s, 2s, 4s, 8s — each with random jitter to avoid thundering herd.
4. Trip the circuit breaker after repeated failures5 consecutive 429s? Stop sending for 30 seconds. This gives the upstream room to recover instead of drowning it in retries.
5. Long-running agent? Use 202 + poll, or SSEAnything past 60 seconds: return 202 Accepted with a job_id and let the client poll, or stream incremental progress over SSE so proxies do not idle the connection out.

Queue producer + resilient worker

# WHAT: Decouple accept from execute
# WHY:  Burst absorption + graceful 429 handling

DEFINE handler(request):
  job_id = ENQUEUE task(request.payload)
  RETURN 202, { job_id }

DEFINE worker_loop():
  LOOP:
    task = PULL from queue
    IF circuit_breaker.is_open:
      REQUEUE task; SLEEP 30s; CONTINUE

    delay = 1
    FOR attempt in 1..5:
      resp = CALL claude(task)
      IF resp.ok: RETURN resp
      IF resp.status == 429:
        SLEEP resp.retry_after OR delay + jitter
        delay = delay * 2
      ELSE: RAISE
    circuit_breaker.RECORD_failure()

Misconceptions

Tokens as they arrive

An agent reply is built one token at a time over 5–30 seconds. Without streaming, the user stares at a blank screen for that whole window, then everything dumps at once. With streaming, the same reply feels alive — tokens flow in, tool calls show as progress chips, and abandonment drops sharply.

The Anthropic SDK gives this for one extra parameter: stream=True in Python or .stream() in Node. You iterate over typed delta events (content_block_delta for text, content_block_start for tool_use, message_stop at the end) and forward each one to the client as a typed SSE event.

The waiter who keeps you posted

BEFORE: You order food and the waiter walks away. Five minutes pass with zero updates. Ten minutes. You start scanning the room wondering if they forgot.

PAIN: Agents take 5–30 seconds. Without progressive output, the user assumes the request died and hits refresh — doubling your API spend and possibly stranding the original generation. Or they abandon entirely. Both outcomes cost real money.

MAPPING: Streaming is the waiter saying "appetizer is on the way" then "entrée is plating now." Each delta event is a small reassurance: a token, a tool-call progress chip, a step-complete badge. The user stays engaged and trusts the system because the system is visibly working.

Claude delta → SSE event → UI update

1. Open Claude in stream modeSet stream=True (Python) or call .stream() (Node). Instead of one final response, you get an async iterator of typed delta events.
2. Translate each delta to a typed SSE eventMap content_block_delta → token, content_block_start with tool_use → tool_start, message_stop → done. Clients render based on event type.
3. Set anti-buffering headersCache-Control: no-cache, Connection: keep-alive, and X-Accel-Buffering: no. Without these, nginx or Cloudflare buffers the entire stream and ruins the UX.
4. Client consumes via EventSourceThe browser's native EventSource parses each event and dispatches it to a handler. Auto-reconnect on drop is built in.
5. Map tool names to friendly labelsShow "Searching filings..." instead of search_filings. The black-box wait becomes a transparent workflow.

SSE chat endpoint

# WHAT: Forward Claude deltas as typed SSE events
# WHY:  Token-by-token UX instead of a 15-second blank screen

DEFINE stream_handler(request):
  SET response.headers:
    Content-Type = "text/event-stream"
    Cache-Control = "no-cache"
    X-Accel-Buffering = "no"

  FOR EACH event IN claude.stream(request.message):
    IF event.type == "content_block_delta":
      YIELD sse("token", { text: event.delta.text })

    ELIF event.type == "content_block_start" AND tool:
      YIELD sse("tool_start", { name: event.tool_name })
      result = RUN tool(event.input)
      YIELD sse("tool_result", { data: result })

    ELIF event.type == "message_stop":
      YIELD sse("done", { usage: event.usage })

# GOTCHA: cap each run at 5 min via a timeout wrapper;
# on timeout, emit a partial event with truncated:true.

Misconceptions

Three gates: authenticate, authorize, throttle

An open agent endpoint is an open door to your LLM budget. One leaked URL and a bot can run up thousands of dollars in API spend, or worse, trigger destructive tool actions. Defense is three sequential gates: authenticate (prove who you are — API key or JWT), authorize (this role is allowed to call this tool), throttle (per-user rate or token budget).

The critical rule: enforce tool permissions in middleware, never in the agent's prompt. Prompt-level rules can be bypassed with prompt injection. Middleware sits in the request path where the model has no influence over the decision.

The hospital with three checkpoints

BEFORE: A hospital where every door is unlocked and anyone can wander into the pharmacy or the surgical suite. No badge readers, no logs, no friction.

PAIN: Disaster. Anyone can grab controlled substances or operate equipment they have no training for. The same logic applies online: an unauthenticated agent API is the open pharmacy — unbounded LLM spend, leaked customer data, destructive tool calls all running on your bill.

MAPPING: Authentication is the badge reader at the front door (does this badge belong to a real person?). Authorization is the access list per room (does this badge unlock this door?). Throttling is the supply cabinet limit (you can take five vials per hour, not five hundred). Three checkpoints, each enforced in the building's wiring — not via "please be polite" signs taped to the wall.

The three-layer defense in flight

1. Extract the credentialRead the Authorization: Bearer <token> header. Missing? Return 401 immediately and skip every later step.
2. Verify the tokenFor API keys: look up in a table. For JWT: verify the signature with the issuer's public key, check expiry. Invalid? 401.
3. Look up the user's role and permissionsFrom the verified token (JWT claims) or a side database. Attach to the request object so downstream middleware can read it.
4. Authorize the requested toolCheck role → allowed tools map. Analyst can call search_filings; only admin can call delete_filing. Forbidden? 403.
5. Check the per-user budgetSliding window in Redis: 20 requests per minute, or N tokens per day. Over limit? 429 with Retry-After.

Auth middleware chain

# WHAT: Three gates before the agent runs anything
# WHY:  Cap LLM spend, block destructive tool calls

DEFINE authenticate(request):
  token = EXTRACT Bearer from headers
  IF missing: RETURN 401
  user = VERIFY(token)   # signature + expiry
  IF invalid: RETURN 401
  request.user = user
  NEXT

DEFINE authorize_tool(tool_name, user):
  allowed = ROLE_TOOL_MAP[user.role]
  # analyst: [search, report]; admin: [+delete]
  IF tool_name NOT IN allowed:
    LOG blocked(tool_name, user.id, user.role)
    RETURN 403
  NEXT

DEFINE throttle(user):
  used = redis.COUNT(user.id, window="60s")
  IF used >= 20:
    RETURN 429 + Retry-After header
  redis.INCR(user.id)
  NEXT

Misconceptions

One question per concept

Tap a card to reveal the answer.