The 4 concepts of tokens

Tap any concept to jump to its 5-card cluster.

• 1What Is a Token?Subword pieces, not words or letters
• 2Context WindowsThe fixed desk the model works on
• 3Prompt BudgetingAllocate the window on purpose
• 4Token Cost AwarenessLocal = free of $, not free of cost

Models read in tokens, not words

A token is a chunk of text from the model's fixed vocabulary — usually a subword. In English, roughly 1 token ≈ 4 characters ≈ ¾ of a word. Common words are one token; rarer ones split ("unbelievable" → un/believ/able).

Everything is measured in tokens: your prompt, the history, the retrieved context, and the generated answer. Context limits and costs are all counted here — so "how long is my prompt?" really means "how many tokens?"

Lego bricks, not letters or whole toys

BEFORE: You'd assume the model reads character-by-character (one key at a time) or word-by-word (like a person reading).

PAIN: Neither is how it works, so estimating length by counting words is off — sometimes badly, especially for code, punctuation, or non-English text, which fragment into many tokens.

MAPPING: Text is built from Lego bricks of a fixed set: some bricks are whole words, some are syllable-sized pieces. "cat" is one brick; "antidisestablishment" is several. To know your length, count the bricks — don't eyeball the words.

Text in, token ids out

1. A tokenizer splits the textIt greedily matches the longest known pieces from a fixed vocabulary (~32k–100k entries).
2. Each piece maps to an idTokens become integers the model actually consumes.
3. Different models, different tokenizersMistral's tokenizer and, say, GPT's tiktoken split the same text into different counts.
4. Count, don't guessUse the real tokenizer to measure — word counts mislead, especially for code and other languages.

Pseudocode

# never eyeball length in words — measure
tokenizer = load_tokenizer("mistral")

n = LEN( tokenizer.encode(text) )   # token count

# rough rule of thumb for English:
#   tokens ≈ chars / 4 ≈ words / 0.75
# but code / JSON / other languages
# fragment into MANY more tokens —
# so always confirm with the real encoder.

Misconceptions

The model's fixed-size working desk

The context window is the maximum number of tokens the model can consider at once — and it includes both your input and the generated output. Mistral-7B typically offers ~8k–32k tokens depending on the build.

Go over the limit and something has to give: the oldest messages fall off, the call errors, or the framework silently truncates. The window is the model's entire short-term working memory — nothing outside it exists to the model.

A desk that only fits so much paper

BEFORE: Imagine working at a desk where you can only see the papers currently laid out on it.

PAIN: The desk is a fixed size. Pile on more documents and older ones slide off the edge — out of sight, out of mind. Anything not on the desk, the worker simply can't use.

MAPPING: The context window is that desk. Prompt, history, and retrieved notes all compete for the same surface, and the answer needs room too. Managing what stays on the desk — and what you file away — is the whole game of context engineering.

Input + output share one limit

1. Everything countsSystem prompt + conversation history + retrieved context + the answer must all fit in the window.
2. Output eats the budget tooIf you want a 1,000-token answer, you must leave 1,000 tokens free after the input.
3. Overflow has consequencesExceed it and you get truncation or an error — and truncated context means dropped facts.
4. Bigger isn't automatically betterAttention cost grows with length, and models can get "lost in the middle" of very long contexts.

What earns a spot on the desk?

# Keep it in-window only if it's...

KEEP the system prompt        # always
KEEP the last few turns       # recent context
KEEP retrieved chunks that
     actually answer the query   # relevance!

DROP / summarize old history
DROP boilerplate & duplicates
RESERVE space for the answer

# relevance > recency > "just in case"

Misconceptions

Spend the window like a budget

Prompt budgeting is deciding, up front, how many tokens each part of the prompt gets: system instructions, conversation history, retrieved context, and reserved space for the answer.

Without a plan, history and context grow until they crowd out the answer or blow the limit. With a plan, you cap each section, trim or summarize when it overflows, and always keep room for the model to respond.

Planning a trip budget

BEFORE: When you book travel, you don't just assume you'll have enough money and start spending.

PAIN: If you blow it all on flights, there's nothing left for the hotel or food — and the trip falls apart at the worst moment.

MAPPING: You add up flights + hotel + food + activities before confirming, and adjust to fit the total. Prompt budgeting is the same: allot tokens to system + history + context + answer, and trim until the whole trip fits the window.

Allocate, then enforce

1. Reserve the answer firstDecide max output tokens and subtract them from the window before anything else.
2. Give each section a capSystem prompt (small, fixed), history (rolling), retrieved context (top-N chunks).
3. Trim when over budgetSummarize old turns, drop low-relevance chunks, cut duplicates — before you hit the limit.
4. Measure and verifyCount tokens of the assembled prompt; confirm input + reserved output ≤ window.

Pseudocode

WINDOW       = 8000
reserve_out  = 1000            # room for the answer
budget      = WINDOW - reserve_out

used  = count(system_prompt)
parts = [system_prompt]

FOR chunk IN [recent_history..., context...]:
  IF used + count(chunk) > budget:
    BREAK              # or summarize instead
  parts.append(chunk); used += count(chunk)

ASSERT used + reserve_out <= WINDOW
send(parts, max_tokens=reserve_out)

Misconceptions

Local is free of dollars, not free of cost

Running Mistral locally through Ollama has no per-token dollar charge — a big reason this course is open-source. But tokens still cost something: compute time, latency, memory, and context space.

A bloated 6,000-token prompt is slow and crowds the window even when it's "free." So the token discipline you'd use to save money on a hosted API still pays off locally — in speed and headroom instead of dollars.

The all-you-can-eat buffet

BEFORE: The buffet is a flat price — refills don't cost extra, so it feels like everything is free.

PAIN: But you still have a finite appetite and a finite afternoon. Pile the plate too high and you're slow, uncomfortable, and out of room for what actually mattered.

MAPPING: Local tokens are buffet refills — no dollar cost. But time (latency), capacity (the window), and energy (compute) are still finite. Take what you need, not everything you can, and you stay fast and roomy.

What tokens really cost locally

1. LatencyMore input to process and more output to generate = a slower response, token by token.
2. Compute & memoryLonger context uses more RAM/VRAM and CPU/GPU cycles — and can push a small machine to swap.
3. Context spaceEvery wasted token is one less available for relevant content and the answer.
4. Hosted contrastOn a paid API those same tokens are literal dollars — the habit transfers directly when you switch providers (M22).

Trim tokens even when they're free

IF the prompt is slow or near the window:
  → trim aggressively
     - summarize old history
     - retrieve fewer, better chunks
     - drop boilerplate / repeated context

IF latency is fine AND window has room:
  → don't over-optimize — clarity first

# On a HOSTED model, always trim:
# there every token is money.

Misconceptions

Test the core insight

One question per concept. Tap to reveal the answer.