The 6 levers of cost optimization

Tap any concept to jump to its 5-card cluster.

• 1Understanding CostsCompute + latency + electricity
• 2Model SelectionRight-size the tier to the task
• 3QuantizationQ4 vs Q8 vs F16
• 4Prompt EfficiencyFewer tokens per step
• 5Batch InferenceConcurrency with a safe cap
• 6Response CachingExact-match + semantic

"Free" local models still cost you

There is no per-token invoice from a cloud provider — but a local model still consumes hardware, electricity, and time. The real equation is cost = inference_time × compute_per_sec + latency_impact.

On a laptop, compute-per-second is near zero but latency impact is huge: a 70B model on CPU can take 10 minutes per query, so your agent loop waits 10 minutes per step. On a rented GPU the compute cost is real money. You optimize for both time and dollars.

Baking your own bread

BEFORE: You run a bakery and stop buying wholesale bread. "Making our own is free!" you announce — no more per-loaf invoice from the supplier.

PAIN: But in-house baking needs a commercial oven, electricity 24/7, a baker's hours, flour storage, and cleaning. You are not paying per loaf — you are running a factory with real overhead that never appears on a single tidy bill.

MAPPING: Open source LLMs work the same way. "No per-token fee" hides the oven (GPU), the power bill (electricity), and the baker's time (latency your users wait). Per-unit cost can win at scale, but the overhead is very real — you must measure it.

Three cost dimensions to track

1. Hardware costAmortized purchase price or hourly rental of the machine running the model.
2. Electricity costA 300W GPU server for 8 hrs/day is roughly $0.07–$0.15/day at US rates.
3. Latency impactSlower tokens/sec means longer agent loops, worse UX, fewer requests served per hour.
4. Measure, then decideThe same model runs 3–6 tok/s on CPU vs 80–120 on a GPU — a 3–10x swing that changes everything.

Pseudocode

FUNCTION true_cost(model, tokens):
  tps      = measured_tokens_per_sec(model)
  seconds  = tokens / tps
  compute  = seconds * compute_cost_per_sec
  # latency hurts even when compute is free
  latency  = seconds * user_pain_factor

  RETURN compute + latency

# laptop CPU: compute~0, latency huge
# rented GPU: compute real, latency small

Misconceptions

Right-size the model to the task

Model choice is a three-way balance: task complexity (how much reasoning does this really need?), model size (how much VRAM it eats), and latency budget (the max acceptable response time).

The most common waste in local deployments: running a 13B–70B model on a task a 7B handles at ~97% quality — burning 2–5x the compute for no measurable gain. Start at the balanced tier (mistral:7b) and only climb when quality actually demands it.

Renting a moving truck

BEFORE: You need to move a single armchair across town, so you book the biggest 26-foot moving truck on the lot — "bigger is safer," you figure.

PAIN: The giant truck guzzles fuel, is a nightmare to park, and crawls through traffic. The armchair rattles alone in a cavern of empty space. You paid triple and arrived later than a hatchback would have.

MAPPING: A 70B model on a classification task is that 26-foot truck. Match the vehicle to the cargo: a 2B model for routing, a 7B for the everyday agent loop, and the big rig only when you are genuinely hauling a house.

The model tier ladder

1. Tier 1 — fastest / smallestphi3:mini, gemma2:2b (2–3 GB). Great for classification and routing.
2. Tier 2 — balanced defaultmistral:7b, llama3.1:8b (4–5 GB). The recommended starting point for most agents.
3. Tier 2B — reasoning specialistdeepseek-r1:7b (4 GB). Stronger multi-step reasoning at a similar size.
4. Tier 3 — near GPT-4mixtral:8x7b, llama3.1:70b (26–40 GB). GPU required; reserve for the hardest tasks.

Match model to task

IF classify / route (<1s budget):
  → phi3:mini or gemma2:2b

IF summarize / extract (<5s):
  → mistral:7b Q4

IF normal agent loop (<10s):
  → mistral:7b or llama3.1:8b

IF planning / code gen (<20s):
  → deepseek-r1:7b

IF complex reasoning (GPU ok):
  → mixtral:8x7b / llama3.1:70b

Default to Tier 2; climb only when a benchmark proves you need to.

Misconceptions

Shrink the weights, keep the smarts

Quantization lowers the numeric precision of a model's weights — from 16-bit floats down to 8-bit or 4-bit integers. That shrinks the file and speeds up inference, at a small cost in quality.

The sweet spot is Q4_K_M: about 98% of F16 quality at ~30% of the memory and roughly 2x the speed. The goal is not maximum fidelity — it is the minimum fidelity your task actually requires.

JPEG for model weights

BEFORE: A photographer shoots RAW — 40 MB per photo, perfect in every detail. Flawless, but a 10,000-photo trip fills a 400 GB drive and loads painfully slowly.

PAIN: You want to post them on a site that loads in under a second. RAW is unusable; JPEG at 20% quality turns faces into muddy blur. The trick is the sweet spot — compression the eye can't detect, at a fraction of the size.

MAPPING: Quantization is JPEG for weights. F16 is the RAW file (perfect, huge). Q4_K_M is the 80%-quality JPEG that looks identical but loads instantly. The "K" and "M" mean the most sensitive layers are compressed more carefully, so quality holds.

The precision ladder

1. Q2_K — extreme squeeze~85% quality, noticeable degradation. Only for severe memory limits.
2. Q4_K_M — the sweet spot~98% quality, ~4 GB VRAM, 2x faster than F16. The default for nearly everything.
3. Q6_K / Q8_0 — high fidelity~99%+ quality, more memory. Use for code gen or evaluation baselines.
4. F16 — reference100% quality, ~14 GB. Fine-tuning and research only.

Which quant to pull?

DEFAULT:
  → pull Q4_K_M   # ~98%, 2x speed

IF only 4 GB RAM total:
  → Q3_K_M

IF code gen / hard reasoning:
  → Q6_K

IF benchmarking a baseline:
  → Q8_0

IF fine-tuning / research:
  → F16 (reference precision)

Unless you have a specific reason to go higher, Q4_K_M wins.

Misconceptions

Fewer tokens = faster steps

Tokens are the unit of work. In the cloud, cutting tokens saves money; locally it saves time — effectively the same thing. An agent that re-appends the full tool response every step compounds fast.

5 steps × 2,000-token context = 10,000 tokens processed. Trim each step to 500 tokens and you process 2,500 — a 4x throughput gain on identical hardware. Lean prompts are free speed.

Packing a carry-on

BEFORE: For a weekend trip you throw your entire wardrobe into a giant suitcase — "just in case I need it."

PAIN: Now you lug a 50-pound bag through every terminal, pay overweight fees, and wait at baggage claim while carry-on travelers walk straight out. Every extra item you'll never wear slows down the whole trip.

MAPPING: Every token you send is an item in the model's suitcase — it re-reads all of it, every step. Pack only what this step needs: trim tool output, cap the response, prune old turns. A light bag moves faster through every checkpoint.

Five token-reduction techniques

1. Stable prefix cachingKeep the system prompt identical — the runtime reuses its cached state for free.
2. Cap response lengthSet a per-step token budget (e.g. 256) so generation can't run away.
3. Truncate tool resultsA 10 KB JSON blob becomes 500 chars of essentials before it hits context.
4. Slide the windowKeep the system prompt + last N turns; drop the oldest to stop context bloat.
5. Ask for JSON, not proseStructured output is 40–60% denser than sentences.

Pseudocode

FUNCTION lean_step(history, tool_out):
  # 3: shrink tool output first
  tool_out = truncate(tool_out, 500)
  history.append(tool_out)

  # 4: keep system + last N turns
  history = system + last_n(history, 6)

  # 1 stable prefix + 2 cap + 5 json
  RETURN model.generate(
    prompt   = history,
    max_out  = 256,
    format   = "json")

Misconceptions

Run many requests at once

For async workloads — document pipelines, nightly enrichment, bulk entity resolution — you don't have to finish one inference before starting the next. The runtime queues concurrent requests against the loaded model.

Sending 4–8 requests at once gives near-linear throughput gains, up to the model's concurrency limit. Push too far and you get queue congestion and out-of-memory risk instead of speed.

Lanes at the grocery checkout

BEFORE: A store opens one checkout lane and forces every shopper through it, one at a time, no matter how long the line grows.

PAIN: The queue snakes to the back of the store. But open 100 lanes with only 8 cashiers and it's chaos — half-staffed registers, confused shoppers, nothing actually moving faster.

MAPPING: Serial inference is the single lane. Batching opens more lanes — but only as many as you have cashiers (GPU/CPU capacity). Four to eight lanes is the sweet spot; a hundred just jams the aisles and risks a memory meltdown.

Batching with a safe cap

1. Fan out the workLaunch all prompts as async tasks instead of awaiting each in turn.
2. Cap concurrencyA semaphore (or p-limit pool) holds active requests at 4–8, never more.
3. Gather resultsCollect responses as they finish; each carries its own timing and error state.
4. Watch the curvebatch=8 is ~6x serial; batch=16 adds little and risks OOM — diminishing returns.

Pseudocode

limit = semaphore(8)   # safe cap

FUNCTION one(prompt):
  WITH limit:            # wait for a slot
    RETURN model.generate(prompt, max_out=256)

FUNCTION batch(prompts):
  tasks = [ one(p) FOR p IN prompts ]
  RETURN await gather(tasks)

# 20 prompts, cap 8 -> ~6x vs serial

Misconceptions

The fastest inference is none

If you've answered a question before, don't pay to answer it again. A two-layer cache handles two cases: identical queries (exact-match on a hash key) and semantically equivalent ones (same meaning, different words).

Together they can eliminate 20–60% of inference calls in repetitive workloads — common in document pipelines and customer chatbots. A cache hit returns instantly; the model never runs.

A chef's prep station

BEFORE: Every time an order comes in, a cook chops onions and simmers stock from scratch — even for the dish that's been ordered forty times tonight.

PAIN: The kitchen falls hopelessly behind. Identical orders each trigger the same ten minutes of prep, and the tickets pile up while diners wait for work that was already done an hour ago.

MAPPING: A cache is the prep station holding ready components. Exact match = the identical dish, plated instantly. Semantic match = "the same thing, hold the salt" — close enough to reuse the base. Only genuinely new orders go to the stove.

Two layers plus TTL

1. Layer 1: exact matchHash the query + model; a dict lookup returns instantly on identical repeats.
2. Layer 2: semantic matchEmbed the query, compare cosine similarity to past ones; > 0.95 is a hit.
3. TTL expiryEach entry has a time-to-live so stale answers age out (e.g. 1 hour).
4. Fall back to inferenceOn a miss, run the model, then store the result in both layers.

Pseudocode

FUNCTION infer(query):
  # layer 1: exact match
  hit = exact_cache[hash(query)]
  IF hit AND NOT expired(hit):
    RETURN hit

  # layer 2: semantic match
  near = vector_search(query, top=1)
  IF near.similarity > 0.95:
    RETURN near.response

  # miss -> run model, then store
  out = model.generate(query)
  store(query, out, ttl=3600)
  RETURN out

Misconceptions

Test the core insight

One question per concept. Tap to reveal the answer.