The 7 concepts of RAG

Tap any concept to jump to its 5-card cluster.

• 1The Knowledge ProblemWhy Claude can't answer about your private data
• 2What Is RAG?The end-to-end search-then-generate pipeline
• 3EmbeddingsTurning text into searchable coordinates
• 4Chunking StrategiesSlicing documents the right way
• 5Vector DatabasesWhere embeddings live and how they're searched
• 6CitationsClaude's native sentence-level provenance
• 7Multimodal InputsPDFs, images, and the Files API

The Knowledge Problem

Every LLM ships with a fixed training cutoff: a date after which it has seen nothing. It also never saw your private files — your wiki, contracts, support tickets, last week's incident report. Ask about any of those and you get either a refusal or a confident fabrication.

Prompt engineering cannot fix this. No matter how clever your wording, Claude can't reference a document it has never seen. The only solutions are: retrain (expensive, slow, brittle), or feed the relevant text into the prompt at query time. RAG is the second path.

The 2023 doctor

BEFORE: Picture a brilliant doctor who graduated in 2023. They studied hard, passed every board, and remember everything from their training.

PAIN: A patient asks about a breakthrough drug approved six months ago. The doctor either says "I don't know" — or, worse, confidently recommends the outdated 2023 protocol that has since been superseded. The patient cannot tell the difference.

MAPPING: Claude is the doctor. Your private docs and recent data are the journals published since 2023. RAG is the assistant who slides the right journal article onto the desk before each diagnosis — so the doctor stops guessing from memory.

Why prompts alone don't help

1. Training freezes the weightsThe model's "memory" is whatever was in the training set as of the cutoff date. Nothing added after that date exists in those weights.
2. Private data was never in the setYour internal docs were never published; the model has zero parameters representing them.
3. Asking nicely fabricatesPressed for an answer it doesn't have, the model generates plausible-sounding text that pattern-matches the question. That's hallucination.
4. Only the prompt is mutableYou can't change the weights at runtime, but you CAN inject new text into every prompt — making fresh facts visible at query time.

RAG vs the alternatives

# Question: Claude doesn't know X. What now?

IF X is private OR newer than cutoff:
  IF X fits in context (small + stable):
    PASTE X directly into the prompt
    # cheapest, simplest, no infra

  ELIF X is large + queries are varied:
    USE RAG # this module
    # search the right slice, paste only that

  ELIF domain style/voice must change:
    CONSIDER fine-tune
    # expensive, slow, but reshapes behavior

ELSE:
  JUST ASK Claude # it already knows

Default to RAG. Fine-tuning is rarely worth $5K–$50K when a vector DB and 200 lines of code do the same job for free.

Misconceptions

Search first. Then generate.

RAG is a six-step pipeline that runs in two phases. The setup phase happens once: read your docs, split them into chunks, turn each chunk into a vector, store it. The query phase runs on every question: turn the question into a vector, search for the nearest chunks, paste them into Claude's prompt, and let Claude write the answer using that context.

The model never changes — only the prompt does. Done well, RAG drops factual hallucination from ~35% to under 3% on private-data questions, and lets Claude cite its sources. The craft is in the chunking, retrieval, and grounding instructions.

The open-book exam

BEFORE: A closed-book exam — the student answers purely from memory. They crammed for months but no human can hold thousands of facts perfectly.

PAIN: When the question lands on something they didn't study deeply, they don't say "I don't know" — they confidently guess. That confident guess is exactly hallucination.

MAPPING: RAG turns the same exam into open-book. Claude is the student, your document corpus is the textbook on the desk, and the retriever is the index at the back of the book — flipping to exactly the right page before each answer instead of guessing from memory.

The six-step pipeline

First four run once at ingest. Last two run on every question.

1. LoadRead raw documents (PDFs, markdown, HTML, transcripts) from disk or an API.
2. ChunkSplit each doc into ~500-character pieces with a 50-char overlap so a fact straddling a boundary survives in at least one chunk.
3. EmbedConvert each chunk into a vector (often 1,536 numbers) that captures its meaning.
4. StoreIndex those vectors in a vector database. Now you can ask "which chunks are nearest to this vector?" in milliseconds.
5. RetrieveAt query time, embed the user's question with the SAME model and ask the DB for the top 3–5 nearest chunks.
6. GenerateBuild the prompt: grounding instruction + retrieved chunks + the question. Send to Claude. Done.

Pseudocode

Two passes — ingest once, query many:

# INGEST (run once when docs change)
FOR EACH doc IN documents:
  chunks = SPLIT(doc, size=500, overlap=50)
  FOR EACH chunk IN chunks:
    vec = EMBED(chunk)
    vector_db.STORE(vec, chunk, source=doc.path)

# QUERY (every user question)
FUNCTION ask(question):
  q_vec = EMBED(question)
  top = vector_db.SEARCH(q_vec, k=3)

  context = JOIN(top, format="[Source N] {chunk}")
  prompt = "Answer using ONLY this context. "
         + "Cite [Source N]. If not in context, say so."
         + context + question

  RETURN CALL Claude(prompt)

Always include the grounding instruction. Without it, Claude may blend retrieved facts with its training memory — defeating the point.

Misconceptions

Embeddings = meaning as numbers

An embedding is a long list of floats — typically 1,536 numbers — that represents the meaning of a piece of text. Similar meanings produce similar vectors, so you can compare two texts by measuring the angle between their vectors instead of comparing words.

This is why "heart attack treatment" and "myocardial infarction therapy" land near each other even though they share zero keywords. It's also why traditional LIKE '%term%' search misses obvious matches: keyword search compares characters, embedding search compares meaning.

Map coordinates for ideas

BEFORE: Picture a world map. Cities have GPS coordinates — Paris and London are close, Paris and Sydney are far. Distance on the map mirrors distance in the real world.

PAIN: Words don't naturally have coordinates. "King" and "queen" feel related, but how would a computer know? Keyword search can't see it — the strings share no characters.

MAPPING: An embedding model assigns every piece of text a coordinate in a 1,536-dimensional "meaning map." "King" and "queen" land near each other; "king" and "spreadsheet" land in different neighborhoods. Search becomes "find the nearest map points to this question."

From text to coordinates

1. TokenizeBreak text into tokens (words or sub-words) the model can process.
2. Run the embedding modelA specialized neural net (smaller than Claude) reads the tokens and outputs a fixed-size float array — typically 1,536 numbers.
3. Store the vectorSave it next to the original text. The numbers carry the meaning; the text is what you'll show the user.
4. Compare with cosine similarityTo compare two texts, take the angle between their vectors. Smaller angle = closer meaning. Score ranges 0 to 1; 1 = identical meaning.

Two sentences with no shared words can score 0.95 if they mean the same thing. That's the entire trick.

Pseudocode

# Convert text into a 1536-dim vector
FUNCTION embed(text):
  RETURN embedding_model.encode(text)
  # output: [0.023, -0.841, 0.129, ... 1533 more]

# Measure similarity between two vectors
FUNCTION cosine_sim(a, b):
  RETURN dot(a, b) / (norm(a) * norm(b))
  # 1.0 = identical, 0.0 = unrelated, -1 = opposite

# Find nearest texts to a query
FUNCTION nearest(query, corpus, k=3):
  q = embed(query)
  scored = [(cosine_sim(q, embed(t)), t) FOR t IN corpus]
  RETURN sort_desc(scored)[:k]

In production, you don't re-embed the corpus per query — you embed once at ingest, store the vectors, and only embed the query at runtime.

Misconceptions

Slice the right way, or retrieval breaks

Chunking is the act of splitting documents into smaller pieces before embedding. You don't embed a whole 5,000-word PDF at once: the resulting vector becomes a blurry average of every topic in the doc, and queries match nothing precisely.

The right chunk size is a Goldilocks problem. Too big and one relevant sentence drowns in 1,999 irrelevant words. Too small and you lose the surrounding context that makes the sentence meaningful. Most teams start at 300–500 characters with 10–20% overlap and tune from there.

The flashcard problem

BEFORE: You're making study flashcards from a 300-page textbook. Before you start writing, you have to decide how much text goes on each card.

PAIN: An entire chapter per card and you re-read pages to find one fact. A single sentence per card and "the treatment was effective" makes no sense without knowing which treatment, which patient, which condition.

MAPPING: Chunks are flashcards. Each one is what the retriever hands Claude on a hit. Right size depends on your content's structure and density — legal text wants smaller chunks, narrative prose wants larger.

Three chunking strategies

1. Fixed-sizeSplit every N characters or tokens, with overlap. Simple and predictable, but cuts mid-sentence and ignores topic boundaries.
2. RecursiveTry natural boundaries first — split on headings, then paragraphs, then sentences. Respects document structure; better than fixed-size for most prose.
3. SemanticCompare embeddings of consecutive paragraphs and split where the topic shifts. Slowest to compute, best retrieval quality on mixed-topic docs.
4. Always add overlap10–20% of chunk size duplicates a few sentences at boundaries, so a fact straddling two chunks survives in at least one.

Pseudocode

# Fixed-size chunking with overlap
FUNCTION chunk(text, size=500, overlap=50):
  chunks = []
  step = size - overlap
  start = 0
  WHILE start < LENGTH(text):
    end = MIN(start + size, LENGTH(text))
    chunks.APPEND(text[start:end])
    start = start + step
  RETURN chunks

# Recursive: try natural splits first
FUNCTION recursive_chunk(text, max_size=500):
  FOR sep IN ["\n\n# ", "\n\n", ". "]:
    parts = text.SPLIT(sep)
    IF ALL parts fit max_size:
      RETURN parts
  RETURN chunk(text, max_size)  # fallback

A 2,000-char doc with size=500 overlap=50 yields ~5 chunks. Each step advances 450 chars; overlap raises chunk count slightly but prevents mid-sentence cuts.

Misconceptions

A database for "what does this mean?"

A vector database stores high-dimensional vectors and finds the closest matches to a query vector extremely fast. Each row holds four fields: an id, the vector (what gets searched), the document text (what gets returned), and metadata (filters and provenance).

Where SQL excels at exact lookups (WHERE name = 'Acme'), vector DBs excel at similarity ("things like Acme"). Most production RAG systems run both: a vector DB for retrieval, a SQL DB for users, transactions, and business logic.

The library reorganized by topic

BEFORE: A traditional library shelves books alphabetically by author. To find "machine learning in healthcare," you'd need to know every author who ever wrote on the topic and check each shelf individually.

PAIN: You miss relevant books because you didn't know the author's name. You waste hours browsing shelves organized by a criterion (author) that has nothing to do with what you actually care about (the topic).

MAPPING: A vector DB is a library shelved by what books are about. All books on "machine learning in healthcare" end up near each other regardless of author. You find them by describing the topic instead of knowing an exact title or keyword.

ANN search — fast at scale

1. You embed the queryThe DB takes your question text, runs it through the same embedding model used at ingest, gets a query vector.
2. HNSW navigates a layered graphInstead of comparing the query against every stored vector, it skims a sparse top layer to find the rough neighborhood, then descends layer by layer to refine.
3. Returns top-K by cosine similarityYou ask for "the 3 nearest" and get back the document text, metadata, and similarity score for each.
4. You never touch raw vectorsThe DB embeds, searches, and unpacks for you. Your code only sees text and metadata.

HNSW does ~200 comparisons instead of 100,000 for the same answer — that's how 100K vs 100M vectors makes barely any difference to query time.

Pseudocode

# Pick a vector DB (ChromaDB, Pinecone, pgvector)
db = vector_db.CONNECT()
collection = db.CREATE("my_docs")

# Insert chunks (vector auto-embedded)
FOR EACH chunk IN chunks:
  collection.ADD(
    id    = chunk.id,
    text  = chunk.text,
    meta  = { source: chunk.source, page: chunk.page }
  )

# Query with optional metadata filter
hits = collection.QUERY(
  text   = user_question,
  k      = 3,
  filter = { source: "filing_guide.md" }
)
# hits = [(text, metadata, score), ...]

For learning: ChromaDB (local, no API key). For production scale: Pinecone (managed) or pgvector (extends Postgres).

Misconceptions

Provenance as a first-class API feature

RAG returns chunks plus an answer, but stitching them — "which sentence in the answer came from which chunk?" — is your code's job, and it's fragile. Claude can reword, summarize, or merge sources, and matching back to the original passages is error-prone.

Anthropic's Citations feature solves this at the API level. You pass source documents into the request as document content blocks; Claude returns its answer with explicit citations[] arrays attached to each sentence, telling you exactly which document and which character span supports each claim. No regex, no fuzzy matching, no fabricated quotes.

Footnotes the editor wrote

BEFORE: A research paper without footnotes. Claims are made; you have to take the author's word for it. Fact-checking means re-reading every cited source from scratch.

PAIN: If you ask the author to add footnotes after the fact, they may misremember which paragraph supported which claim, or worse, invent a citation that sounds right. Hallucinated citations are a real problem when you ask Claude to "include sources like [1] [2]."

MAPPING: Citations move footnoting from author-after-the-fact to structured at write-time. Claude can't fabricate a citation because the API forces every claim to point at a real character span in a real document you provided.

From document blocks to cited spans

1. You attach documentsUp to ~20 per request, each with a unique title, passed as content blocks with "type": "document" and citations: enabled.
2. Claude reads themThe model treats them as the authoritative source set. It still answers your question; the docs are the only allowed evidence.
3. Each sentence gets a citation arrayThe response includes citations[] on each text block: document_index, document_title, and the exact cited_text span used.
4. You render with provenanceDisplay Claude's text alongside footnote markers that link to the actual source span — audit-grade, not "best effort."

RAG vs Citations — pick per job

# Pick by corpus size and provenance need

IF corpus is huge (millions of docs):
  USE RAG
  # vector search to narrow down first

IF candidate set is small (<20 docs):
  USE Citations
  # sentence-level provenance, audit-grade

IF regulated domain (legal, medical, finance)
   AND corpus is huge:
  USE RAG + Citations # the strongest combo
  shortlist  = vector_db.SEARCH(query, k=10)
  CALL Claude(documents=shortlist, citations=on)
  # scale of vector search,
  # audit-grade provenance of Citations

Anti-pattern: asking Claude in the prompt to "include citations like [1] [2]" and parsing them out. Numbers can be fabricated. Use the structured Citations API.

Misconceptions

PDFs, images, and the Files API

Real-world documents arrive as PDFs with embedded tables, scanned forms, charts, and screenshots — not clean plain text. Claude has three native primitives for these without a separate OCR step or vision model: PDF support, vision, and the Files API.

Knowing which to reach for — and when to combine them with your RAG pipeline — closes the last gap between a toy chatbot and a production document agent. They all use the same content-block paradigm, so swapping inputs is a one-line change.

One inbox, many envelope types

BEFORE: Imagine an office that handles paper letters, faxes, photos, and parcels — but routes each to a different building, with different handlers and different turnaround times. Cross-referencing a fax with a photo takes weeks.

PAIN: The customer doesn't care about envelope types. They want one answer. Every routing handoff is a place where context gets lost or the wrong handler picks up the wrong document.

MAPPING: Multimodal inputs let Claude be a single inbox. Text, PDF, image, and Files-API references all go in one content array; Claude reasons across all of them in one pass. No OCR pipeline, no separate vision service, no cross-system stitching.

Four content block types

1. type: "text"The user message or prompt — the standard text block.
2. type: "document"A PDF up to ~100 pages and ~32MB. Claude reads it natively, including tables and headings. Combine with citations for audit-grade provenance.
3. type: "image"PNG, JPEG, GIF, or WebP. Useful for charts, screenshots, scanned forms, and diagrams. Tokens scale with image dimensions.
4. file_id referenceUpload once via the Files API, get a file_id, reference it in many requests. Beats re-encoding the same PDF in every call.

All four block types travel in the SAME content array. Claude reasons across them jointly — one PDF + one screenshot + one question, in one call.

Pick the right modality

# Match input type to scenario

IF one-off PDF <100 pages:
  USE document block (base64) + citations

ELIF same PDF in 50+ requests:
  USE Files API + reference by file_id
  # upload once, save bandwidth

ELIF PDF >100 pages OR >32MB:
  USE RAG to shortlist
       + document blocks for top-K pages

ELIF screenshot, chart, scanned form:
  USE image block
  # tokens scale with dimensions; resize first

ELIF mixed (PDF + screenshot + text):
  USE all three in one content array
  # Claude reasons across them jointly

Misconceptions

One question per concept

Tap a card to reveal the answer.