The 5 concepts of shipping an agent

Tap any concept to jump to its 5-card cluster.

• 1Agent as HTTP ServiceStateless endpoint any caller can reach
• 2Streaming with SSEPush steps as they happen
• 3Async Job QueueFire-and-poll for long work
• 4Validation & Rate LimitingReject bad input & noisy callers
• 5Docker DeploymentRuns identically everywhere

Wrap the agent behind an HTTP endpoint

An agent that only runs in your notebook can be called by exactly one person: you. Wrapping it as an HTTP service turns it into something any caller — a UI, another service, a cron job — can invoke over the network.

The key design choice is to make each endpoint stateless: the server holds no memory between requests. Every call carries everything the agent needs (the query plus a session ID to look up history). That makes scaling trivial — any replica can serve any request — and lets many callers reach the agent at once.

The analyst in the locked room

BEFORE: Imagine a brilliant analyst who lives in a locked room. The only way to work with her is to walk in, hand her a sheet of paper, and wait for the answer to slide back under the door.

PAIN: A second team can't get in while you're inside. Automated pipelines are blocked on a human gatekeeper. And she remembers nothing, so every visit is a fresh handshake. The analyst is great; the access model is broken.

MAPPING: The HTTP API is an intercom bolted to the door. Any caller speaks through it without entering. Each call is independent, so two callers reach her at once, and the session ID on each message is the label on the envelope — the caller owns the memory, not the analyst.

One request, end to end

1. Caller sends a requestA POST with the query, an optional session ID, and config — plus an auth token in the header.
2. Server authenticatesCheck the API key. No key, or wrong key, gets a clean 401 before the agent ever runs.
3. Agent runs statelesslyThe server loads any prior history by session ID, runs the loop, holds no state itself.
4. Return a typed responseResult plus metadata (session ID, iterations, latency) as a documented JSON contract.

Pseudocode

DEFINE endpoint POST "/agent/run":
  # reject unauthenticated callers first
  IF not valid_api_key(request.header):
    RETURN 401 "unauthorized"

  body = parse(request.body)
  history = load_session(body.session_id)

  result = run_agent(body.query, history)

  # server keeps NO state between calls
  RETURN 200, {
    result:     result.text,
    session_id: body.session_id,
    latency_ms: elapsed(),
  }

Misconceptions

Push each step as it happens

A synchronous call makes the client wait for the whole agent run — 45 seconds of a blank spinner. Streaming flips that: the server pushes one event per step of the loop — a thought, a tool_call, then a final result — and the client renders each the moment it arrives.

The mechanism is SSE (Server-Sent Events): a one-way stream of text chunks over a single long-lived HTTP connection. It needs no WebSocket upgrade, passes through every proxy, and is ideal for showing agent progress in real time.

Kitchen updates vs. a silent wait

BEFORE: You order at a restaurant and the kitchen goes silent. No word for 45 minutes — you can't tell if they're cooking, lost your order, or closed.

PAIN: The silence is agonizing and risky. You might give up and leave, or the waiter's one-shot "here's your meal" arrives so late the connection (your patience, or an HTTP proxy) has already timed out.

MAPPING: SSE is a waiter who keeps you posted: "starters are on," "mains searing now," "plating." Each update is one event down a single open line. You see steady progress, so the wait feels short and nothing times out from silence.

An event per loop step

1. Open the streamServer responds with text/event-stream and keeps the connection alive instead of closing it.
2. Emit typed eventsEach step writes event: thought then data: {json}, ended by a blank line (double newline).
3. Client reads progressivelyIt splits on the blank line, parses each event, and renders it the instant it arrives.
4. Close on the resultA final result (or error) event ends the stream cleanly.

Pseudocode

DEFINE stream endpoint POST "/agent/stream":
  set header "Content-Type: text/event-stream"
  set header "Cache-Control: no-cache"  # or proxy buffers

  FOR EACH step IN run_agent_steps(query):
    type = step.type        # thought | tool_call | result
    emit("event: " + type)
    emit("data: " + to_json(step))
    emit("")               # blank line ENDS the event

  # on error, emit one final event: error
  CLOSE stream

Misconceptions

Fire the job, poll for the result

Streaming handles agents under a minute. But a bulk task processing 10,000 records can run for many minutes — longer than any HTTP connection or proxy will hold open. The fix is an async job queue.

The server accepts the request, returns 202 Accepted with a job ID instantly, and runs the work in the background. The client then polls GET /jobs/{id} until it sees status: complete. This breaks the request-response link entirely, so work can run for minutes or hours with zero timeout risk.

The dry-cleaner ticket

BEFORE: You drop off a huge pile of laundry and the clerk says "wait here at the counter until it's all clean." You'd stand there for hours, blocking the line behind you.

PAIN: Nobody waits at a counter for hours. The shop can't serve other customers past you, and if you step away for a second you lose your place entirely — exactly what a proxy timeout does to a long request.

MAPPING: Instead the clerk hands you a numbered ticket and you leave. The 202 response is that ticket. You come back (poll) and show the number; if it's ready you collect it, if not you check again later. The shop keeps serving everyone while your order runs in back.

Submit, run, poll

1. Submit → 202 + job IDPOST enqueues the job, stores it as pending, and returns a job ID immediately.
2. Background worker runs itA worker picks up the job, flips it to running, then complete or failed.
3. Client polls the jobGET /jobs/{id} every couple of seconds; unknown IDs return 404 so clients stop polling.
4. Collect the resultOnce status is complete, the stored result is returned in the poll response.

Pseudocode

DEFINE POST "/agent/jobs":
  job_id = new_id()
  store[job_id] = { status: "pending" }
  enqueue_background(run_job, job_id, body)
  RETURN 202, { job_id, poll_url }

DEFINE run_job(job_id, body):      # background
  store[job_id].status = "running"
  TRY:
    store[job_id].result = run_agent(body)
    store[job_id].status = "complete"
  CATCH e: store[job_id] = { status:"failed", error:e }

DEFINE GET "/jobs/{id}":
  IF id not in store: RETURN 404
  RETURN store[id]

Misconceptions

Reject bad input and greedy callers

An open endpoint is an open door. Two guards protect it. Validation checks every request against a typed schema before the agent runs — empty query, oversized payload, wrong type all get a clean 422 instead of a crash deep in the loop.

Rate limiting stops one noisy caller from eating all your inference capacity. A token bucket gives each API key a bucket of tokens that refills at a steady rate; each request spends one, and when the bucket is empty the caller gets 429 Too Many Requests with a Retry-After header.

The nightclub door

BEFORE: Picture a club with no door staff. Anyone walks in — people without ID, people in banned outfits, and a handful who barge back in every ten seconds.

PAIN: The wrong crowd floods the floor. Guests who don't meet dress code cause chaos inside, and the repeat-entry crowd hogs the bar so real patrons can't get served. The club (your agent) is overwhelmed by who got through the door.

MAPPING: Validation is the dress-code check — malformed requests are turned away at the door (422), never reaching the floor. The token bucket is the bouncer's tally: each guest gets a set number of entries that replenish slowly; over that, you wait outside (429, Retry-After).

Two gates before the agent

1. Validate the schemaCheck types and bounds — query length, iteration caps. Fail fast with 422 and a clear message.
2. Identify the callerDerive a key from the API token (or IP). This is who the rate limit applies to.
3. Refill & spend a tokenAdd tokens earned since last call (capped), then try to spend one for this request.
4. Allow or rejectToken available → run the agent. Bucket empty → 429 with Retry-After telling them when to come back.

Pseudocode

# GATE 1 — validate against the schema
IF not schema.valid(body):
  RETURN 422, schema.errors

# GATE 2 — token bucket per caller
key    = caller_id(request)
bucket = buckets[key]  # {tokens, last_seen}

elapsed = now() - bucket.last_seen
bucket.tokens = min(CAP, bucket.tokens + elapsed * REFILL)
bucket.last_seen = now()

IF bucket.tokens < 1:
  RETURN 429, header "Retry-After"
bucket.tokens = bucket.tokens - 1
PROCEED to run agent

Misconceptions

Package once, run identically everywhere

"Works on my machine" fails the moment someone else clones the repo. Docker packages your service — code, dependencies, runtime, exact versions — into an image that runs the same on your laptop, a CI runner, and a cloud server.

For a local model you run the inference engine as a sidecar container next to your API container. A compose file wires them on a private network so the model is never exposed to the internet, and the API only starts once the model container reports healthy. The whole stack becomes one compose up command.

The shipping container

BEFORE: Before standardized containers, dockworkers loaded ships piece by piece — barrels, crates, sacks — each needing custom handling. Move cargo to a different port or truck and everything had to be repacked.

PAIN: Every transfer was slow, error-prone, and unrepeatable. What loaded fine in one port jammed in the next, because nothing about the packing was standard.

MAPPING: The shipping container fixed this: seal your goods once, and every crane, ship, and truck handles the identical box. A Docker image is that box — sealed with your app and its exact dependencies, it lands and runs the same on any machine that speaks Docker.

From image to running stack

1. Build a lean imageA multi-stage build compiles deps in a fat builder, then copies only the artifacts into a slim runtime — smaller, safer.
2. Add a health checkThe image declares how to test readiness, so the orchestrator knows when it's truly up.
3. Compose the servicesOne file defines the API + model sidecar, a private network, secrets, and a persistent model volume.
4. Order the startupThe API waits on the model's health check, so it never accepts traffic before the model can answer.

What goes where in the stack?

IF deps need build tools (compilers):
  → use a multi-stage build
     (fat builder, slim runtime)

IF a service must be ready before
   another starts:
  → add a healthcheck +
     depends_on: service_healthy

IF the model must not be public:
  → put it on a private network,
     don't publish its port

IF data must survive a restart
   (models, job state):
  → store it in a named volume

# secrets = env vars, never baked in

The same compose file that runs locally deploys to any host with Docker — only TLS, secrets, and the model's listen address change.

Misconceptions

Test the core insight

One question per concept. Tap to reveal the answer.