The 6 concepts of cost optimization

Tap any concept to jump to its 5-card cluster.

• 1The Cost Anatomy of an Agent CallWhere the money actually goes — six line items
• 2Caching StrategiesPrompt cache, response cache, embedding cache
• 3Model RoutingCheapest capable model wins — classify and dispatch
• 4Token OptimizationCompress prompts, summarize history, top-3 RAG
• 5Extended ThinkingPay for reasoning when retries cost more than thinking
• 6Batch API50% off for anything that can wait until tomorrow

Where the money actually goes

An agent call has six cost line items: input tokens, output tokens, tool API calls, embedding/retrieval, compute overhead, and — the silent killer — multi-turn history multiplication. Most teams watch input tokens and miss the two real drivers: output costs 3–5× more per token, and every turn re-sends the full conversation history.

Cost grows geometrically, not linearly. You cannot optimize what you cannot see, so the foundational move is to instrument tokens_in, tokens_out, model, cache hits, and total cents on every trace — then attack the largest line item.

The restaurant bill

BEFORE: You order an entree expecting to pay the menu price. Easy.

PAIN: The bill arrives with six lines: entree (input), dessert at 5× the per-ounce rate (output), tax (compute), sommelier tip for bottles you barely sipped (tools), the appetizer sampler (embeddings), and a "per-visit surcharge that doubles each visit" (history). Your $12 dinner cost $90.

MAPPING: Every agent call carries those same six surcharges. Output is the dessert — small in volume, huge in price. History is the per-visit charge — doubles every turn. Read the whole bill, not just the entree.

The six line items, ranked by surprise factor

1. Input tokensSystem prompt + history + tool results + user message. ~$3/MTok on Sonnet. Cheap per token, but volume grows every turn.
2. Output tokens (5× rate)What Claude generates. ~$15/MTok on Sonnet — five times input. Verbose responses are disproportionately expensive.
3. Multi-turn multiplicationTurn 10 sends ~20K input tokens because history is re-sent every call. Cumulative input by turn 10 is ~55× a single turn.
4. Tool executionThird-party APIs (search, DB, weather). Billed by the provider, not Anthropic.
5. Embeddings & retrievalPer-query embedding + vector DB lookup. Small per call, large at high QPS — especially if you re-embed the same text.
6. Compute overheadYour servers, Redis, log pipeline, observability. Mostly fixed, still on the bill.

Instrument first. Once you see which line item bleeds, the right lever is obvious.

Find the bleeding line item, then act

# Look at last 24h of cost data, by line item.
# Whichever is >40% of total — attack that one first.

IF system_prompt_tokens > 30% of total_input:
  USE prompt caching # 90% off the cached prefix

ELIF output_tokens > 30% of total_cost:
  SHRINK output # max_tokens, "respond concisely",
                # structured JSON instead of prose

ELIF history_tokens grow geometrically by turn:
  SUMMARIZE old turns # sliding window

ELIF a single model handles everything:
  ROUTE simple traffic to Haiku

ELIF workload tolerates > 1h latency:
  SUBMIT via Batch API # 50% off

ELSE:
  RE-MEASURE — you don't know what's bleeding yet.

Don't optimize what isn't expensive. The biggest line item moves the bill the most.

Misconceptions

Three layers of caching, three kinds of repeated work

An agent does the same work over and over: re-sends the same system prompt every call, re-answers the same FAQ from scratch, re-embeds the same documents. Each repetition is a paid round-trip you don't need. Caching at three layers — prompt cache, response cache, embedding cache — eliminates the redundancy without changing agent behaviour.

The highest-leverage layer is Anthropic's prompt cache: tag your stable prefix (system prompt, tool defs, few-shot examples) with cache_control and cached input tokens drop to ~10% of normal price. First call pays full price plus a small write fee; every call within the 5-minute TTL pays the discount. The TTL resets on each hit, so steady traffic keeps the cache warm indefinitely.

Sunday meal-prep

BEFORE: You cook every meal from scratch. Breakfast: chop onions, boil water, season, cook. Lunch: chop the same onions, boil water again. Dinner: same.

PAIN: Three hours of cooking daily for ingredients you already chopped once. Identical prep work, billed three times a day.

MAPPING: An uncached agent re-sends the identical 4,000-token system prompt every request, re-answers the same support question, re-embeds the same product description. Sunday meal-prep is the three-layer cache — prep the system prompt once (prompt cache), store finished answers (response cache), keep chopped vegetables in the fridge (embedding cache). Same meals, fraction of the work.

The three cache layers in order

1. L1 — Prompt cache (Anthropic native)Mark your stable prefix with cache_control: {type: "ephemeral"}. First call pays a write fee; subsequent calls within 5 min pay ~10% of input price. TTL resets on every hit.
2. L2 — Response cache (semantic)Embed the query, search a vector store of recent (query, response) pairs. Cosine > 0.93 → return the cached response directly — 0% LLM cost. Best for FAQs and status checks.
3. L3 — Embedding cache (Redis LRU)Cache the vector for any text you've already embedded. Hot embeddings serve in <1ms vs a 50–200ms API call.
4. Tool-result cache (bonus)For deterministic tools (SEC filing, exchange rate), wrap the tool with the same L1/L2/L3 stack so identical calls don't pay twice.

Stack checked top→bottom. Each hit short-circuits the rest.

Pseudocode

Three-layer cache with prompt-cache marker on the stable prefix:

FUNCTION answer(user_query):
  # L2: semantic response cache — full short-circuit if hit
  q_vec = EMBED(user_query)
  IF hit := response_cache.SEARCH(q_vec, threshold=0.93):
    RETURN hit.response                    # 100% LLM savings

  # L3: embedding cache for any RAG lookups we still need
  context = rag.FETCH(user_query, embed_cache=redis_lru)

  # L1: Anthropic prompt cache on the stable prefix
  msg = {
    "system": [{
      "type": "text",
      "text": SYSTEM_PROMPT + TOOL_DEFS,
      "cache_control": {"type": "ephemeral"}   # 90% off
    }],
    "messages": history + [{"role": "user",
                            "content": user_query + context}]
  }
  resp = claude.messages.create(model, msg)

  # Promote into L2 for next time
  response_cache.SET(q_vec, resp, ttl=1h)
  RETURN resp

The single line cache_control: ephemeral usually delivers the biggest cost win in the whole stack.

Misconceptions

Send each request to the cheapest model that can do the job

Production traffic isn't uniformly hard. Roughly 60–70% of agent requests are simple — greetings, FAQ lookups, status checks, format conversions. Sending those to Sonnet (or worse, Opus) is paying neurosurgeon rates for band-aids. Model routing classifies each request and dispatches it to the smallest capable model: Haiku for simple, Sonnet for standard, Opus only for genuinely hard reasoning.

The classifier itself is cheap — a short Haiku call (~$0.0004) or an embedding-similarity comparison (~$0.00001). Routing savings dwarf the classifier cost by 100–1000×. Done well, routing alone cuts agent bills 30–50% with no quality loss, because simple tasks don't need a big model.

The hospital triage nurse

BEFORE: A hospital that sends every walk-in — paper cut, broken arm, brain tumor — straight to the chief neurosurgeon. Brilliant. $5,000/hour. One of them.

PAIN: Bankruptcy by paper cut. Brain-tumor patients can't get appointments because the neurosurgeon is bandaging knees.

MAPPING: The triage nurse changes everything. Paper cut → nurse practitioner (Haiku). Standard care → GP (Sonnet). Surgical case → neurosurgeon (Opus). Same outcomes, fraction of the cost. The classifier is the triage nurse — trivial cost compared to misrouting every patient to the most expensive provider.

Classify, dispatch, fall back

1. ClassifyHaiku call ("simple / standard / complex?") or embedding similarity vs pre-labeled clusters. Tier label in <100ms.
2. DispatchSimple → Haiku ($1/$5 per MTok). Standard → Sonnet ($3/$15). Complex → Opus ($15/$75). Reserve Opus for measured-hard cases.
3. Run with the chosen modelSame prompt, tools, caching — just a different model name. The rest of the stack is unchanged.
4. Detect failure, fall back upIf a Haiku response is short, evasive, or fails a validator, retry on Sonnet. Log to retrain the classifier.
5. Cap your worst caseKeep an Opus budget per user/day. If exceeded, force route to Sonnet and surface a friendly degrade message.

A 3% misclassification with auto-fallback beats a 0% misclassification all-Sonnet bill every time.

Pick a model in three questions

# For each incoming request, answer in order:

IF request matches simple pattern:
  # greetings, yes/no, format conversion, status check,
  # FAQ lookup, single-field extraction
  USE Haiku                 # 3x cheaper than Sonnet

ELIF request needs multi-step reasoning OR tool use OR
     code generation OR summarization:
  USE Sonnet                # the workhorse default

ELIF request needs deep judgment, nuanced synthesis,
     or you have measured Sonnet failing on this class:
  USE Opus                  # 5x Sonnet — reserve carefully

ELSE:
  DEFAULT to Sonnet         # when in doubt, the workhorse

# Always wire a fallback: if the cheap model output fails
# validation (too short, refusal, malformed JSON), retry one
# tier up and log for classifier retraining.

Most teams don't need an Opus tier at all. Start with two tiers (Haiku + Sonnet); add Opus only when you measure Sonnet failing.

Misconceptions

The cheapest token is the one you never send

Most prompts are bloated. The system prompt is full of filler ("be helpful, be thorough, be polite") the model does by default. History is re-sent in full every turn, including small-talk from twenty turns ago. RAG pulls back ten chunks when two were relevant. The model reads all of it — and you pay.

Token optimization is four moves: compress the system prompt, cap output, summarize old turns, top-k RAG only. Together they shrink a 30,000-token turn-10 request to under 9,000 — ~70% fewer tokens, often with better answer quality because distraction was removed along with cost.

Packing a carry-on instead of a moving truck

BEFORE: Long weekend trip. You actually need ~20 pounds of clothes and toiletries.

PAIN: You book a moving truck instead and throw in 47 shirts, 12 pairs of shoes, your bookshelf, and a kitchen blender "just in case." You pay per pound. You wear three outfits the entire trip.

MAPPING: An unoptimized agent is the moving truck — 4,000-token system prompt with filler, every turn re-sent verbatim, ten RAG chunks when two are relevant. The carry-on is a compressed prompt, sliding-window history, and top-k retrieval. Same trip, fraction of the bill.

Four token cuts, in priority order

1. Compress the system promptStrip filler ("be helpful and thorough"), collapse duplicate rules, switch verbose paragraphs to numbered lists. Typical 4,000 → 1,500 tokens with no quality loss. A/B test the compressed version against the original.
2. Cap outputOutput costs 5× per token — the highest-leverage cut. Use max_tokens. Ask for structured JSON instead of prose. Add "respond concisely" to the system prompt.
3. Summarize old conversation turnsSliding window: keep the last 3–5 turns verbatim, compress everything older into a 200-word rolling summary. A 20-turn chat drops from ~40K to ~8K input tokens — and breaks the geometric growth curve.
4. Top-k RAG, not top-everythingRetrieve top 2–3 most-relevant chunks, not 10. Add a similarity threshold (e.g., cosine > 0.8) so weak matches are filtered out before they reach the model. Less noise, lower cost, often better answers.

Order matters: output cap is highest leverage per minute of work. System-prompt compression is highest leverage per request.

Pseudocode

The token-trim pipeline that runs before every LLM call:

FUNCTION build_optimized_request(user_msg, history, rag_hits):

  # 1. Compressed system prompt (one-time work, ship once)
  system = COMPRESSED_SYSTEM_PROMPT      # 1500 tok, was 4000

  # 2. Sliding-window history with rolling summary
  IF len(history) > 5:
    summary = SUMMARIZE(history[:-5], max=200)  # cheap Haiku call
    trimmed_history = [{"role": "system",
                        "content": "Earlier: " + summary}]
                      + history[-5:]
  ELSE:
    trimmed_history = history

  # 3. Top-k RAG with similarity threshold — no padding
  context = [hit FOR hit IN rag_hits[:3]
                 IF hit.cosine > 0.8]

  # 4. Output cap — the 5x lever
  RETURN claude.messages.create(
    model="sonnet",
    system=system,
    messages=trimmed_history + [{"role": "user",
                                 "content": context + user_msg}],
    max_tokens=512                       # cap output, was 4096
  )

Each line is independently testable. Toggle one at a time and measure the savings.

Misconceptions

Pay for reasoning, not for length

Sometimes the right lever is spending more tokens on the right thing. Anthropic's extended thinking mode lets Sonnet or Opus produce a private reasoning trace before the final answer. You set a token budget; the model uses up to that budget privately, then writes its public response. Thinking tokens bill at the output rate.

On simple tasks this is wasteful. On hard ones (multi-step proofs, multi-file refactors, complex planning), the alternative is a buggy multi-turn loop where you pay for retry after retry. Extended thinking turns several wasted attempts into one careful answer. Match it to task difficulty — not "always on" or "always off".

Wider category: Anthropic’s extended thinking is one flavor of reasoning models — a 2025 trend. OpenAI’s o-series (o3, o4-mini), DeepSeek R1, and Google Gemini Deep Think do the same thing differently: separate model family vs parameter, full trace vs summary, explicit budget vs effort levels. Same idea: spend internal tokens to think before answering.

Carpenter rule: measure twice, cut once

BEFORE: A carpenter is asked to cut a custom shelf for a recessed alcove. Shop across town. Expensive lumber.

PAIN: Eyeball it. Shelf is 1″ too short. Drive back, buy more lumber. Cut again: too long. Three trips, three boards, half a day lost.

MAPPING: Extended thinking is the carpenter taking five minutes with a tape measure before the first cut. Small upfront cost (thinking tokens), no cascade of retries. On a 2×4 cut in half, the tape measure is overkill. On the alcove shelf, it pays for itself ten times over.

Budget reasoning, then answer

1. Opt in per requestAdd thinking: {type: "enabled", budget_tokens: 4000} to the Messages call. Same Sonnet/Opus model — not a separate variant.
2. Model reasons privatelyUses up to the budget producing an internal trace (planning, weighing options, self-checking). Trace is not streamed.
3. Model emits the final answerVisible response is generated after the budget is consumed. Trace is inspectable but not user-facing.
4. You're billed for thinking + outputBudget bills at the output rate (~$15/MTok on Sonnet). A 4,000-token budget adds ~$0.06 per request.
5. A/B test the impactSame task with vs without thinking. Measure success rate. Ship thinking only where the gain pays for the budget.

Latency increases too — bad for streaming UX, fine for offline workflows.

When does thinking pay off?

# Per-request decision — not a global default.

USE extended thinking WHEN:
  · multi-step math, logic, or proofs
  · multi-file code refactors with dependency graph
  · complex planning where one wrong step cascades
  · self-checked extraction (validate JSON before commit)
  · A/B test shows success rate lifts >= 15%
                                  # enough to pay for budget

SKIP extended thinking WHEN:
  · simple lookups, format conversions, summaries
  · tasks Haiku already nails
                                  # wrong axis — route, don't think
  · streaming UX (thinking is not streamed)
  · latency-sensitive paths (thinking adds seconds)

DEFAULT:
  thinking OFF                       # opt-in, not opt-out

Anti-pattern: thinking on every request "to be safe." That just inflates cost on tasks that don't need it.

Misconceptions

50% off for anything that can wait until tomorrow

Not every workload needs an answer in two seconds. Nightly evals, bulk document classification, periodic report generation, retroactive scoring — none need real-time latency. Anthropic's Batch API accepts up to 100,000 requests per submission and returns results within 24 hours, at ~50% off input and output prices vs the synchronous Messages API.

Better still: batch stacks with prompt caching. A 90% cache discount on top of a 50% batch discount is ~95% effective off cached input. If "tomorrow morning" is acceptable, leaving Batch API on the table is leaving money on the table.

Off-peak electricity tariffs

BEFORE: Your utility charges one flat rate per kWh, 24 hours a day. Dishwasher, dryer, EV charger all draw at peak afternoon prices.

PAIN: Premium rates for loads that don't care when they run. The dishwasher doesn't mind 2pm vs 2am — but it costs the same because you never told the utility you'd time-shift.

MAPPING: The Batch API is the off-peak tariff. Anthropic gives 50% off in exchange for "I need this before morning, not in two seconds." Nightly evals, bulk taggers, weekly digests — all the LLM dishwashers — belong on the discounted rate.

Four steps from JSONL to results

1. Prepare a JSONL fileOne request per line. Each carries a custom_id (your join key) and a standard Messages request body. Up to 100,000 requests per batch.
2. SubmitPOST /v1/messages/batches with the JSONL. Returns a batch ID and in_progress status.
3. Wait (poll or webhook)Anthropic processes async, up to 24 hours. Poll GET /v1/messages/batches/<id> or register a webhook. Status flips to ended.
4. Download resultsOne response per line, each tagged with the original custom_id so you can join back. Bills at the discounted rate.

Note: Batch API doesn't support multi-turn tool calling inside a single request. Blocking workflows still need the synchronous API.

Pseudocode

Submit, poll, collect — with prompt caching stacked for max savings:

# 1. Build a JSONL payload — one request per row
batch_lines = []
FOR EACH row IN docs_to_classify:
  batch_lines.append({
    "custom_id": row.id,                  # your join key
    "params": {
      "model": "claude-sonnet-4-6",
      "max_tokens": 256,
      "system": [{
        "type": "text",
        "text": CLASSIFIER_PROMPT,
        "cache_control": {"type": "ephemeral"}  # stack the discounts
      }],
      "messages": [{"role": "user", "content": row.text}]
    }
  })

# 2. Submit — up to 100K requests per batch
batch = claude.messages.batches.create(requests=batch_lines)

# 3. Wait for completion (poll or webhook)
WHILE batch.status != "ended":
  SLEEP(60); batch = claude.messages.batches.get(batch.id)

# 4. Stream results, joined back via custom_id
FOR EACH result IN claude.messages.batches.results(batch.id):
  store(row_id=result.custom_id, output=result.message)
# Bill: ~50% of synchronous — up to ~95% off cached input

If "by tomorrow morning" is acceptable, batch is the default. The synchronous API is for things a human is actively waiting on.

Misconceptions

One question per concept

Tap a card to reveal the answer.