MODULE 1 OF 30
Track 1 — Foundations
Module

The LLM Mental Model

How Claude actually works under the hood — tokens, inference, sampling, and the "thinker not calculator" mindset every agent builder needs.

⏱ 15 min read 📚 3 concepts 🃏 19 cards
Input tokens Transformer Layers (all at once) Probabilities per token Pick one Autoregressive loop — repeat until done

Swipe left to start →

What you'll learn

3 Concepts — tap to jump

Key insight: Claude doesn't look things up — it predicts tokens. This shapes every design decision you'll make as an agent builder.

Concept 1 of 3
Big Idea

What Is a Large Language Model?

A Large Language Model (LLM) like Claude is a neural network trained to predict the most likely next piece of text — over and over, one token at a time. When you send Claude a question, it doesn't look up an answer in a database. Instead, it computes probabilities for every possible next word, picks the most likely one, appends it, then repeats until the response is complete.

Every impressive thing Claude does — writing code, answering questions, translating languages, reasoning through problems — is a side effect of getting extremely good at one single job: predicting what comes next.

The "Large" refers to two things: the training data (terabytes of text — books, code, web pages, papers) and the number of internal parameters (hundreds of billions of adjustable numbers). More training + more parameters = richer patterns = more useful predictions.

💡 Why It Matters for Agents

When your agent gives a wrong answer, it's because token prediction chose a plausible-but-incorrect continuation — not because Claude is "confused." This distinction shapes how you write prompts, design tools, and build guardrails throughout this course.

Concept 1 of 3
Analogy

The World's Most Well-Read Autocomplete

💡 Everyday Analogy

BEFORE: Old chatbots followed hand-written rules — giant if/then trees. Ask something the programmer didn't anticipate, and you'd get "I don't understand." Ask a search engine a nuanced question, and you'd sift through ten links hoping one had your answer.

PAIN: That approach broke down constantly. Novel questions, subtle phrasings, and creative tasks all required someone to manually program every possible response — an impossible task.

MAPPING: An LLM is the world's most well-read autocomplete. Your phone's autocomplete has read your messages; Claude has read billions of documents — every book, manual, conversation, scientific paper. Instead of following rules, it learned statistical patterns from that mountain of text, so it can handle questions nobody explicitly programmed it for. It's autocomplete that went to every university, read every manual, and practiced every writing style.

The key difference from search: A search engine finds existing text that matches your query. An LLM generates new text by predicting what would naturally come after your prompt — no retrieval, just pattern-based generation.

Concept 1 of 3
How It Works

Token Prediction — Step by Step

  1. 1 Tokenize: Your prompt is split into tokens — roughly word-pieces. "Hello world" → 2 tokens; "antidisestablishmentarianism" → several tokens.
  2. 2 Embed: Each token is converted to a high-dimensional vector — a list of ~thousands of numbers representing its meaning in context.
  3. 3 Attend: Transformer layers let every token "see" every other token simultaneously — this is how context is built.
  4. 4 Predict: The final layer outputs a probability score for every word in the vocabulary (~200K words). Higher score = more likely next token.
  5. 5 Sample & loop: One token is selected based on probabilities, appended to the sequence, and the whole process repeats from step 3 — autoregressive generation.
Input Tokens Transformer Layers Attention + FFN Probability Distribution "Paris" 92% Output Token ← Autoregressive feedback loop → Step 1-2 Step 3 Step 4 Step 5
Concept 1 of 3
Pseudocode

Token Prediction Loop

This is the core algorithm behind every response Claude generates. The entire inference process reduces to this loop:

# What Claude does for every API call GIVEN input_tokens = TOKENIZE(your_prompt) LOOP until stop: embeddings = EMBED(input_tokens) context = ATTENTION(embeddings) # all tokens see each other logits = PREDICT_NEXT(context) # score for every vocab word probs = SOFTMAX(logits / temperature) next_token = SAMPLE(probs) # pick one token input_tokens.APPEND(next_token) IF next_token == END_TOKEN: BREAK IF len(output) >= max_tokens: BREAK RETURN collected output tokens

The "thinker not calculator" insight: Each SAMPLE step is probabilistic. Even the same prompt can yield slightly different tokens. Design agents for "highly consistent," not "bit-for-bit identical." This is why verification and guardrails exist.

Concept 1 of 3
Misconceptions + Takeaway

Common LLM Misunderstandings

❌ "Claude looks up answers in a database"
✅ Claude generates text by predicting tokens — there is no lookup or retrieval step. Correct facts emerge from training patterns, not a database query. This is also why Claude can produce confident but wrong answers.
❌ "Claude understands language the way humans do"
✅ Claude finds statistical patterns in language at a scale that produces useful results. It has no internal model of truth and no beliefs. When it seems to understand, it's pattern-matching very effectively.
❌ "Bigger models are always more accurate"
✅ Larger models are more capable at complex reasoning, but they can hallucinate more convincingly. Size doesn't eliminate the need for verification and guardrails.
💡 Key Takeaway

An LLM is the world's most well-read autocomplete — it predicts tokens, not answers. Every impressive capability emerges from this single mechanism. Understanding token prediction is the foundation for understanding why prompts work, why tools are needed, and why guardrails matter.

Concept 2 of 3
Big Idea

How Inference Actually Works

Every call to the Claude API runs two distinct phases: Prefill (reads your entire prompt in parallel, once) and Decode (generates each output token one-at-a-time, sequentially). This two-phase architecture determines your latency profile.

PREFILL

All input tokens processed at once. Fast but scales with prompt length. Drives time-to-first-token.

DECODE

One token at a time, sequentially. Scales with output length. Drives tokens-per-second.

Streaming doesn't make inference faster — it just shows you each token as it decodes instead of waiting for the full response. Same total time, better perceived latency.

💡 Why This Matters

A 50K-token prompt with a 100-token answer feels slow to start but finishes quickly. A 200-token prompt with a 4000-token answer feels snappy at first but takes time. Knowing which phase dominates helps you optimize the right thing.

Concept 2 of 3
Analogy

Typing a Reply on a Phone

💡 Everyday Analogy

BEFORE: You probably picture an LLM as a function: question goes in, full answer comes out. That mental model is wrong in a specific way that matters once you start optimizing latency and cost.

PAIN: Without the right picture, you can't explain why a 50K-token prompt costs money before Claude writes a single word back, or why the first token arrives 800ms later but subsequent tokens stream out at 60ms each.

MAPPING: Think of someone typing a long reply on their phone. They read everything they've written so far (prefill, runs once), pick the most likely next letter, type it, re-read, pick the next letter, type it — and repeat. There is no "full answer" sitting in the model waiting to be unwrapped. The answer is constructed token-by-token, in real time, in the same API call.

Practical rule: Long prompts → slow first token. Long responses → slow total time. Use streaming for chat UIs so users see progress immediately.

Concept 2 of 3
How It Works

Two Phases of Every API Call

  1. 1 Your prompt arrives: The full message — system, conversation history, current user turn — is tokenized into N tokens.
  2. 2 PREFILL (once): All N tokens processed simultaneously through all transformer layers. The model builds a KV cache storing attention values for each token.
  3. 3 First token ready: Prefill outputs logits for the very first output token. This moment is your Time-To-First-Token (TTFT).
  4. 4 DECODE loop: Sample one token → append to sequence → run just that new token through layers (reusing KV cache) → get next logits → repeat.
  5. 5 Stop: Decode ends when the model outputs an end token or hits max_tokens. Total decode time scales linearly with output length.
PREFILL All prompt tokens, parallel Runs ONCE DECODE One token at a time, sequential Runs for EACH output token Drives time-to-first-token Drives tokens/second
Concept 2 of 3
Pseudocode

The Two-Phase Inference Model

# Phase 1: Prefill — runs ONCE per API call kv_cache = PREFILL(entire_prompt) # Cost: ~O(N²) but parallel. Sets time-to-first-token. first_logits = kv_cache.last_logits # Phase 2: Decode — runs for EACH output token response = [] LOOP until done: next_token = SAMPLE(first_logits, temperature) response.APPEND(next_token) IF next_token == END_TOKEN: BREAK IF len(response) >= max_tokens: BREAK # Reuse cache — O(1) per token kv_cache.UPDATE(next_token) first_logits = TRANSFORMER(next_token, kv_cache) RETURN response

The KV cache is why decode is cheap per-token. Once prefill builds it, each new decode step only processes the single new token — not the entire growing sequence.

Concept 2 of 3
Misconceptions + Takeaway

Inference Misunderstandings

❌ "Inference and training are the same thing"
✅ Training updated billions of weights using gradient descent across millions of examples — it happened at Anthropic before the model shipped. Inference reads those frozen weights to predict tokens. You only ever do inference via the API.
❌ "Streaming is faster than non-streaming"
✅ Same total wall-clock time. Streaming just delivers tokens as they decode instead of buffering them. Use streaming for UX (users see progress immediately). Use non-streaming when you need the full response before parsing (JSON, tool dispatch).
❌ "Big prompts are slow because Claude reads more"
✅ Prefill processes the prompt in parallel, but attention scales roughly O(N²). Long prompts hurt time-to-first-token, not decode speed. Prompt caching (M22) can turn 5s of prefill into 200ms for repeat-heavy prompts.
💡 Key Takeaway

Every API call has two phases: Prefill (fast, parallel, pays once per call) and Decode (sequential, pays per output token). Long prompt → slow start. Long response → slow finish. Streaming doesn't change total time — it changes perceived latency.

Concept 3 of 3
Big Idea

Temperature, Top-p & Top-k

After computing probabilities for every possible next token, Claude must pick one. Temperature controls how spread out the probability distribution is before sampling — a creativity dial from "always pick the safest word" (0) to "let surprising words compete" (1.0). Top-p and top-k narrow the menu of tokens Claude even considers.

0.0
Greedy — always top word
0.5
Balanced — top words compete
1.0
Creative — diverse words

For agent work, you'll almost always want temperature 0.0–0.3. An agent routing 5,000 customer emails per day needs consistent, predictable decisions — not creative variation. Save high temperature for creative writing tasks.

Concept 3 of 3
Analogy

The Restaurant Menu

💡 Everyday Analogy

BEFORE: Without sampling controls, a language model always picks the single highest-probability next word — like a restaurant that only serves its most popular dish, every day, to every customer, no matter what.

PAIN: Every sentence would sound mechanical and repetitive. You'd get stuck in ruts — always the same phrasing, always the most obvious word. Great for a FAQ bot, terrible for anything that needs nuance or creativity.

MAPPING: Temperature is a creativity dial. At 0, Claude always picks the safest, most predictable word — the restaurant's number-one dish. At 1.0, lower-probability but more interesting options get a real shot. Top-p says "only consider words that make up the top 90% of the probability mass." Top-k says "only consider the top 50 most likely words." Together, they give you precise control over how adventurous Claude's word choices are.

Same prompt, three outputs: At temp 0.0 — "The moon is Earth's only natural satellite." At 1.0 — "Our luminous neighbor drifts through the cosmic dark." Same model, same question — different creativity dial setting.

Concept 3 of 3
How It Works

Sampling — From Logits to Token

  1. 1 Logits: The transformer outputs a raw score ("logit") for every word in the vocabulary. Higher score = more likely next token.
  2. 2 Temperature scaling: Divide all logits by the temperature value. Low temp (0.1) makes the top score dominate. High temp (1.0) keeps the distribution spread out.
  3. 3 Softmax: Convert scaled logits into probabilities (all positive, summing to 100%). Now each word has a percentage chance of being picked.
  4. 4 Top-k filter (optional): Keep only the top-k highest-probability tokens; discard the rest. Blocks nonsense tokens in the long tail.
  5. 5 Top-p filter (optional): Keep the smallest set of tokens whose probabilities sum to ≥ p. Renormalize remaining tokens, then sample one.
temp=0.0 92% temp=1.0 30% 22% 15% one dominant choice many choices compete
Concept 3 of 3
Pseudocode

Sampling Algorithm

FUNCTION sample_next_token(logits, temp, top_k, top_p): # Step 1: Temperature scaling scaled = logits / temp # lower T = sharper peak probs = SOFTMAX(scaled) # Step 2: Top-k filter (optional) IF top_k > 0: keep = TOP_K_INDICES(probs, top_k) probs = ZERO_OUT_EXCEPT(probs, keep) probs = RENORMALIZE(probs) # Step 3: Top-p / nucleus filter (optional) IF top_p < 1.0: sorted = SORT_DESCENDING(probs) cumsum = CUMULATIVE_SUM(sorted) cutoff = FIRST_INDEX(cumsum >= top_p) probs = ZERO_OUT_AFTER(probs, cutoff) probs = RENORMALIZE(probs) RETURN RANDOM_SAMPLE(probs)

Agent rule of thumb: Set temperature to 0.0–0.3 for any decision-making agent (routing, tool selection, classification). Use 0.7–1.0 only for explicitly creative tasks like writing or brainstorming.

Concept 3 of 3
Misconceptions + Takeaway

Temperature Misunderstandings

❌ "Temperature 0 means fully deterministic output"
✅ Almost. Temperature 0 uses argmax (no sampling), which is highly consistent. But server-side floating-point rounding and batch scheduling can still cause minor variation across runs. Design for "extremely consistent," not "bit-for-bit identical."
❌ "Higher temperature = better, more intelligent responses"
✅ Higher temperature = more creative and varied responses, not more intelligent ones. For agents routing decisions or selecting tools, high temperature creates unpredictable behavior and makes systems hard to test. Use low temperature for reliability.
❌ "Top-p and top-k are the same thing"
✅ Top-k keeps the N most likely tokens regardless of their probabilities. Top-p keeps the smallest set of tokens whose cumulative probability meets a threshold. Top-p adapts to the distribution; top-k is fixed count.
💡 Key Takeaway

Temperature is the creativity dial — 0 for predictable agents, 1.0 for creative writing. Top-p and top-k narrow the menu before sampling. For agent work, low temperature produces consistent, testable, auditable decisions. Variety is a feature for creative tasks; it's a bug for decision-making systems.

Quick Quiz

Test Your Understanding

Tap each question to reveal the answer.

💻
Ready to make your first API call?

The desktop version of M01 includes a full interactive lab — make your first Claude API call, experiment with temperature, and build a working CLI chatbot. Python and Node.js code, copy-paste ready.

Open Full Module →
← M00: Course Overview M02: Tokens →

Building AI Agents with Claude · Module 1 of 30