The 4 concepts of tokens

Tap any concept to jump to its 5-card cluster.

• 1What Are Tokens?BPE, sub-words, and the ~100K vocabulary
• 2Why Tokens MatterCost, latency, and context limits in one unit
• 3The Context Window200K shared by input + output, no fallback
• 4Token Counting in PracticeEstimation, the usage field, and count_tokens

Tokens are the unit Claude actually reads

A token is a chunk of text smaller than a word. Claude never sees raw letters — every message is sliced into tokens before it arrives, and tokens are what Claude reads and writes one at a time. The model works only with the numeric IDs of those chunks; the human-readable text is just a wrapper at the edges.

Rule of thumb in English: 1 token ≈ 4 characters ≈ 0.75 words. So a 100-word paragraph is ~133 tokens. Common words like "the" stay whole; long or rare words get split; an emoji can cost 2–3 tokens.

Tokens are syllables for AI

BEFORE: Imagine teaching a child to read by showing them whole paragraphs — no letters, no syllables, just walls of text. The child has nothing to grip.

PAIN: Computers face the same problem with raw bytes. "understanding" has no structure to a machine, and there are millions of possible words across every language — far too many to memorize whole.

MAPPING: Tokens solve this the way syllables solve reading. When you read "understanding," your brain breaks it into un-der-stand-ing. The tokenizer breaks it into "understand" + "ing". A fixed ~100K-piece vocabulary covers any text in any language.

Byte-pair encoding, in plain English

1. Scan billions of wordsCount how often every adjacent character pair appears in training text.
2. Merge the most frequent pair"t" + "h" is everywhere — merge into "th" and add to the vocabulary.
3. Repeat thousands of times"th" + "e" → "the". "ing", "tion", "Claude" all emerge as single pieces. Rare strings stay as small fragments.
4. Freeze the vocabularyAfter ~100K merges, you have a reusable kit. Any text in any language encodes as a sequence of these IDs.
5. Encode at runtimeYour message is greedy-matched against the vocabulary — longest pieces win — producing the integer IDs Claude actually reads.

The vocabulary never changes after training. Same model = same tokenizer = same counts, every call.

Pseudocode

How a tokenizer turns text into IDs:

# vocabulary was built once during training: ~100K pieces
FUNCTION tokenize(text):
  ids = []
  i = 0
  WHILE i < length(text):
    # greedy: try the longest matching piece first
    FOR piece IN vocab.sorted_by_length_desc():
      IF text starts_with(piece, at=i):
        ids.append(vocab.id_of(piece))
        i = i + length(piece)
        BREAK
  RETURN ids

# Example
tokenize("Understanding tokens")
# => [48583, 278, 5765, 82]   # 2 words, 4 tokens

Real BPE encoders are smarter than greedy — they pick the merge sequence that produces the fewest tokens overall. The shape is the same.

Misconceptions

One unit drives cost, speed, and capacity

Tokens decide three things you'll babysit constantly: cost (you pay per token, and output costs 3–5x more than input), limits (the 200K context window is measured in tokens), and latency (more tokens means more time per response). Optimize one and the other two improve in step.

This is why token efficiency is a core agent-engineering skill. A bloated system prompt costs more, eats your context window, AND adds latency — every extra token taxes all three dimensions at once.

Tokens are the kilowatt-hour of AI

BEFORE: Picture an electricity bill. You don't pay for "lights" or "fridges" — you pay for kilowatt-hours. One unit covers everything: every appliance, every room, every minute.

PAIN: If you ignore that one unit, you can't reason about anything. You can't compare a fridge to a heater, can't predict next month's bill, can't tell whether a smart-thermostat upgrade will pay back. The whole house is opaque.

MAPPING: Tokens are Claude's kWh. Pricing, latency, and the 200K context window all use the same unit. Once you start counting tokens, every product decision becomes a back-of-the-envelope calculation: "Sonnet at 2K input + 500 output, 50K calls/day, that's ~$700/day — can we trim 20% off the system prompt?"

The three downstream effects

1. CostSonnet 4.5: $3 / $15 per 1M (in / out). Opus 4: $15 / $75. Output is 3–5x more expensive than input.
2. LatencyOutput tokens are produced one at a time, each depending on all prior ones. Twice the output ≈ twice the wall-clock time.
3. Context limitsSystem + history + message + response share one 200K window. Every input token is one Claude can't use for output.
4. Reasoning qualityInformation buried in a sea of tokens gets less attention. Less noise = better answers, on top of cheaper and faster.

A 20% prompt trim usually pays back in cost, latency, AND quality — one of the rare engineering moves with no downside.

Where to cut tokens first

# Goal: cut cost AND latency. Where do you start?

IF output_tokens are large (> 1K per call):
  CAP max_tokens to a sane ceiling
  ADD instructions: "answer in under 200 words"
  # biggest dollar lever — output is 3–5x more expensive

ELIF system_prompt > 1K tokens:
  TRIM verbose instructions, examples, role-play
  CACHE the prompt with cache_control
  # sent every call — trims compound across the conversation

ELIF history is unbounded:
  SUMMARIZE at 60–70% utilization (not 95%)
  DROP stale tool results
  # prevents the silent context-window crash

ELIF input documents are huge:
  USE RAG: retrieve only the relevant slice
  # covered in M09

ELSE:
  SWITCH to a cheaper tier (Sonnet → Haiku)
  IF quality holds.

Order matters: cap output first (highest $ per token), then trim system prompt (sent every call), then manage history (the silent killer).

Misconceptions

One 200K-token desk, shared by everyone

The context window is the total tokens Claude can see in a single API call: system prompt + history + user message + response. All current Claude models share the same 200K limit. Exceed it and the API returns a hard error — no graceful truncation, no automatic summary, no upgrade tier.

Inside the window, recall is perfect. Outside it, Claude has zero memory. That all-or-nothing boundary is why every serious agent eventually grows an explicit memory system on top of the raw window.

The finite desk

BEFORE: Imagine a project where you could spread papers on an infinitely large desk — every email, contract, and note visible at once. You'd never have to set anything aside.

PAIN: Real desks are finite. When yours fills up, the stack topples and your work stalls. Worse, your filing cabinet is in another room — you can't glance at it mid-thought.

MAPPING: The 200K context window is Claude's desk. System prompt, history, your message, AND Claude's response all sit on it at once. Overflow throws an error. And unlike you, Claude has no filing cabinet — whatever isn't on the desk doesn't exist for that call.

What shares the 200K and why it overflows

1. Every call is independentClaude has no built-in memory between calls. Your app re-sends the full history every turn — those tokens count.
2. System prompt is sent every callA 2K-token system prompt over 50 turns = 100K billed tokens. Caching cuts the bill but still consumes context space.
3. History grows monotonicallyEvery user turn, assistant turn, tool call, and tool result joins history. Without management it climbs until crash.
4. max_tokens reserves output spacemax_tokens=4096 means "leave 4096 free for me." If input + max_tokens > 200K, the request fails before Claude generates anything.
5. No graceful fallbackOverflow returns 400 prompt is too long. You manage the budget, or your call dies.

How to budget the 200K

# A sane default split for most agents

system_prompt    = 1K   # cache it; sent every call
tool_definitions = 2K   # cache; rarely changes
history_budget   = 100K # leaves headroom; summarize past this
user_message     = 5K   # typical question + small docs
response_reserve = 8K   # max_tokens for generated answer
safety_cushion   = 84K  # for spikes, retrieved chunks, tool output

# ——— Pre-flight check before every call ———
total_in = count(system) + count(tools) + count(history) + count(message)
IF total_in + max_tokens > 200_000:
  SUMMARIZE oldest history turns
  OR DROP stale tool results
  OR LOWER max_tokens

# Trigger summarization at 60–70%, never wait for 95%
IF total_in > 140_000:
  trigger_compaction()

Position critical info at the START (system prompt) or END (latest user message). The "lost in the middle" effect is real — you'll learn it in M08.

Misconceptions

Estimate fast, count exact when it matters

To manage the budget you must measure it. Two tools cover every case: a back-of-the-envelope estimate (1 token ≈ 4 characters, instant, free, ~10–20% off in code or non-English text), and the SDK's count_tokens endpoint (a real API call, ~50–100ms latency, but exact).

Practical rule: estimate everywhere, then call count_tokens only when your estimate suggests you're above 70% utilization. That's when precision matters and where wrong numbers crash the call.

Pre-flight fuel check

BEFORE: A pilot doesn't fuel a plane by guessing how full the tank looks. There's a calibrated gauge in the cockpit that shows exact remaining fuel before takeoff.

PAIN: Without that gauge, you find out you're short on fuel mid-flight — way too late. Same with tokens: discovering you're over budget means the API call has already failed and your user is staring at an error.

MAPPING: Estimation is the dipstick — quick, rough, fine for parking. count_tokens is the calibrated gauge — slower, but trusted before every flight. Use the dipstick at the gas station, the gauge before takeoff.

Three sources of token counts

1. The 4-char estimateDivide character count by 4. Free, instant, no network. Off by 10–20% in code or non-English text. Fine for cheap sanity checks.
2. The usage field on every responseEvery successful API response carries usage.input_tokens and usage.output_tokens. Free with the call you already made — perfect for live spend dashboards and post-hoc audits.
3. The count_tokens endpointA dedicated SDK call that returns exact input-token count for a system prompt + message list, BEFORE you send the real request. Adds ~50–100ms of latency — budget for it.
4. The budget formulaCombine them: available_for_response = 200,000 − system − history − message. Cap max_tokens at available_for_response and you'll never crash on overflow.

In production, you almost always combine #1 (cheap gate) with #3 (precise check above 70%) and log #2 for billing.

Pseudocode

Pre-flight budget guard for every Claude call:

# --- Cheap path (estimate) ---
FUNCTION estimate_tokens(text):
  RETURN length(text) / 4

# --- Exact path (SDK call) ---
FUNCTION exact_tokens(system, messages):
  RETURN claude.count_tokens(
    model="claude-sonnet-4-6",
    system=system,
    messages=messages
  ).input_tokens

# --- Pre-flight guard (use before every API call) ---
FUNCTION safe_call(system, messages, want_max_out):
  rough = sum(estimate_tokens(m.text) FOR m IN messages)

  IF rough > 140_000:           # above 70% — precision time
    used = exact_tokens(system, messages)
  ELSE:
    used = rough                  # estimate is fine

  available = 200_000 - used
  IF available < want_max_out:
    messages = compact(messages)  # summarize / drop
    RETURN safe_call(system, messages, want_max_out)

  RETURN CALL claude.messages.create(
    system=system, messages=messages,
    max_tokens=min(want_max_out, available)
  )

Always read response.usage.input_tokens and output_tokens after the call — that's your billing source of truth.

Misconceptions

One question per concept

Tap a card to reveal the answer.