The 4 concepts of the LLM model

Tap any concept to jump to its 5-card cluster.

• 1What Is an LLM?A next-token predictor, not a lookup
• 2The Model ZooBase, instruct, chat, embeddings, vision
• 3How Inference WorksAttention + one token at a time
• 4Temperature, Top-p & Top-kThe knobs that shape every answer

A machine that predicts the next token

A large language model is, at its core, a next-token predictor. Given the text so far, it outputs a probability for every possible next token (a word-piece) and picks one. Repeat, and you get sentences.

Nobody programmed its answers. It was trained on an enormous amount of text and learned the statistical patterns of language — grammar, facts, styles, reasoning shapes. That's why it can answer things no one wrote a rule for, and also why it can confidently make things up: it's predicting plausible text, not looking up truth.

From hardcoded phone tree to learned patterns

BEFORE: Old chatbots were a giant phone tree — a human wrote every branch: "if the user says X, reply Y." Say anything off-script and it broke.

PAIN: Language is infinite; you can't enumerate every question. Hand-writing rules for each one is impossible, so rule-based bots were rigid and shallow.

MAPPING: An LLM never got the branches. It read billions of examples and learned the patterns of how text continues. Ask something new and it predicts a plausible continuation — flexible and open-ended, but grounded in likelihood, not a lookup table of facts.

Train once, predict forever

1. Training (once)The model reads a huge corpus and adjusts billions of weights to get better at predicting the next token.
2. Frozen weightsAfter training, the weights don't change. Running the model (inference) only reads them — it doesn't learn from your chat.
3. Prediction (every call)Given your prompt, it computes next-token probabilities and samples one.
4. Repeat to generateAppend the token, feed it back, predict again — until a stop signal. That loop is the whole show.

When is an LLM the right tool?

# Reach for an LLM when...

IF the task is language-shaped
   (summarize, classify, rewrite, converse)
   AND a few wrong answers are tolerable:
  USE an LLM

ELIF you need an exact, always-correct
      computation (math, lookups, totals):
  USE code / a database
  # or give the LLM a TOOL that does it

ELSE:
  # combine: LLM to decide, tools for truth

The whole course is about that last line: LLM for judgment, tools for facts.

Misconceptions

"AI" is a zoo, not one animal

"LLM" isn't one thing. A base model just continues text. An instruct model is tuned to follow commands. A chat model is tuned for back-and-forth turns. And there are specialists: embedding models (text→vectors), vision models (images), code models.

Size matters too: a 7B model like Mistral runs on a laptop; a 70B needs a serious GPU. Picking the right species — and the smallest one that does the job — is half of building efficient agents.

One word, many species

BEFORE: When people say "AI," they picture one thing: a chatbot that takes a question and gives an answer.

PAIN: That single image hides real differences. Handing a raw base model a chat prompt, or using a giant model where a tiny one would do, wastes money and produces confusing results.

MAPPING: "AI model" is like the word "vehicle" — it covers a bicycle, a truck, and an ambulance. You wouldn't take a bicycle to move furniture. Knowing the species (base vs instruct vs chat, small vs large, text vs vision) lets you pick the right vehicle for the trip.

The main species

1. BaseRaw next-token continuation. Great for completion, awkward for instructions — it just keeps writing.
2. Instruct / ChatFine-tuned to follow commands and hold a conversation. This is what you use for agents (Mistral-Instruct).
3. EmbeddingTurns text into a vector of numbers for search and similarity — the engine behind RAG (M09).
4. Vision / CodeSpecialists for images or programming. Same core idea, different training diet.

Which model for the job?

IF building a conversational agent:
  USE an instruct/chat model  # e.g. Mistral

IF you need search / similarity:
  USE an embedding model      # not a chat model

IF the input is images:
  USE a vision model          # e.g. LLaVA

# Then pick the SMALLEST size that passes
# your evals — bigger = slower + costlier.

Default to the smallest model that clears your quality bar; scale up only when it fails.

Misconceptions

Read all at once, write one at a time

Inference is running the trained model to generate text. A transformer reads your entire prompt in parallel (that's what "attention" enables), building a rich picture of context all at once.

But it writes one token at a time: predict the next token, append it, then re-read the whole thing and predict again. This is why responses stream in word by word, and why longer outputs take longer — each token is a fresh forward pass.

Skimming the whole page vs reading letter by letter

BEFORE: Older models (RNNs) read text strictly in order, one word after another, carrying a running summary — like sounding out a sentence letter by letter.

PAIN: Reading serially is slow and forgetful: by the end of a long sentence, the beginning has faded, so long-range connections get lost.

MAPPING: A transformer skims the whole page at once and lets every word "look at" every other word (attention). Nothing fades, and it runs in parallel — then it composes the reply one token at a time, always re-reading the full page before the next word.

One generation step

1. TokenizeYour prompt is split into tokens and turned into vectors.
2. Attention over everythingEvery token attends to every other, in parallel, building context-aware representations.
3. Predict next tokenThe model outputs a probability for every possible next token.
4. Sample & appendPick one token (see Concept 4), append it, and loop — re-reading the whole sequence each time until a stop token.

Pseudocode

# autoregressive generation = a loop
tokens = tokenize(prompt)

WHILE TRUE:
  probs = model(tokens)         # attention over ALL tokens
  next  = sample(probs)         # pick one (temperature...)
  IF next == END_OF_TEXT:
    BREAK
  tokens.append(next)           # feed it back in
  stream(next)                  # word appears on screen

# each token = one full forward pass ->
# longer output = proportionally slower

Misconceptions

Knobs that turn probabilities into a pick

At each step the model has a probability for every next token. Sampling controls decide how to choose. Temperature reshapes the distribution: low (0–0.3) sharpens toward the top choice (focused, repeatable); high (0.8+) flattens it (varied, creative).

Top-k keeps only the k most likely tokens; top-p (nucleus) keeps the smallest set whose probabilities sum to p. Both trim the long tail of unlikely tokens before sampling — guardrails against nonsense.

The one-dish diner vs the tasting menu

BEFORE: With no sampling controls, the model always picks the single highest-probability token — a diner that only ever serves its #1 dish.

PAIN: Always-the-top is repetitive and can get stuck in loops. Always-random is chaos — it'll serve you something inedible from the bottom of the menu.

MAPPING: Temperature is how adventurously you order; top-k/top-p is trimming the menu to only the good dishes first. Low temperature + tight top-p = reliable favorites; higher = a varied tasting menu, still from the dishes worth eating.

From probabilities to a token

1. Model outputs probabilitiesOne score per possible next token.
2. Temperature reshapes themLow sharpens toward the top; high flattens toward even odds.
3. Top-k / top-p trims the poolDrop the unlikely tail so nonsense can't be chosen.
4. Sample oneDraw from what's left. Temperature 0 = always the top token (as deterministic as the backend allows).

Pick temperature by task

IF extraction, classification, JSON,
   tool-calling, anything needing
   repeatable / exact output:
  temperature = 0 – 0.2      # focused

ELIF brainstorming, drafting,
      creative writing:
  temperature = 0.7 – 1.0    # varied

ELSE:
  temperature = 0.3 – 0.5    # balanced default

Agents mostly want low temperature — you're wiring behavior, not writing poetry.

Misconceptions

Test the core insight

One question per concept. Tap to reveal the answer.