The 7 concepts of RAG

Tap any concept to jump to its 5-card cluster.

• 1The Knowledge ProblemCutoffs, private data, hallucination
• 2What Is RAG?The end-to-end pipeline
• 3Embeddings from ScratchMeaning as coordinates
• 4Chunking StrategiesSplitting docs the right way
• 5Vector DatabasesFast search by meaning
• 6CitationsVerifiable provenance
• 7Multimodal InputsPDFs, images & OCR

The model doesn't know your docs

Every model has a fixed knowledge cutoff — a date after which it saw no data. It also never saw your private company docs, your recent research, or your internal databases. Ask about any of it and the model hallucinates: confident, plausible, wrong.

Prompt engineering can't fix this. No matter how clever the wording, the model can't reference a document it has never seen. You have to hand it the source.

The doctor who stopped reading journals

BEFORE: Imagine a brilliant doctor who graduated in 2023 and has never opened a journal since — practicing entirely from what they learned in school.

PAIN: A patient asks about a breakthrough approved six months ago. The doctor either says "I don't know" or confidently recommends an outdated protocol — and the patient can't tell the difference.

MAPPING: That's the cutoff problem. The model is the doctor, your fresh docs are the new journals, and RAG is the system that drops the right journal on the desk exactly when it's needed.

Why the gap exists

1. Training freezesThe model learns from data up to a cutoff date, then stops. Nothing after that exists to it.
2. Private data is invisibleYour internal docs were never in the training set — the model has zero access to them.
3. Gaps get filled with guessesAsked about the unknown, the model produces fluent, plausible text that may be flatly wrong.
4. RAG closes the gapRetrieve the relevant document at query time and paste it into the prompt — no retraining.

RAG or fine-tune?

IF knowledge changes often
   or lives in private docs:
  → use RAG (retrieve at query time)

IF you need new behavior / style
   / a skill, not new facts:
  → consider fine-tuning

# RAG: minutes, ~free, no ML expertise
# fine-tune: weeks, $5k-$50k, retrain
# most doc problems = RAG

A support team cut hallucinated answers from ~35% to under 3% with a RAG prototype built in an afternoon.

Misconceptions

Search, then generate

RAG is two phases. A one-time setup phase: chunk your docs, embed each chunk, store the vectors. Then a query phase that runs on every question: search for the most related chunks and paste them into the prompt so the model answers from real text.

The user never sees the machinery. They ask a question and get an accurate, sourced answer — because the model read actual retrieved chunks instead of guessing from memory.

Closed book vs open book

BEFORE: Picture a closed-book exam where you must answer purely from memory — you crammed for months, but thousands of facts simply wouldn't fit.

PAIN: Hit a topic you didn't study deeply and you either leave it blank or guess. Guessing confidently is exactly what a hallucination looks like.

MAPPING: RAG turns it into an open-book exam: the model is the student, your corpus is the textbook on the desk, and retrieval is the index at the back that flips to the right page instead of guessing.

The six-stage pipeline

1. Load & ChunkRead your documents and split them into small pieces (setup, once).
2. Embed & StoreTurn each chunk into a vector and save it in a vector database (setup, once).
3. RetrieveEmbed the user's question, search the store, take the top-K closest chunks.
4. GeneratePaste those chunks into the prompt; the model answers and cites its sources.

Load → Chunk → Embed → Store → Retrieve → Generate

Pseudocode

# SETUP (once)
FOR each doc:
  chunks = split(doc)
  FOR each chunk:
    store(embed(chunk), chunk)

# QUERY (every question)
FUNCTION answer(question):
  q_vec = embed(question)
  top = store.search(q_vec, k=3)
  prompt = context(top) + question
  RETURN model.generate(prompt)

Misconceptions

Meaning becomes geometry

An embedding turns text into a list of numbers — a point in space — where the numbers encode what the text means, not which words it uses. Two sentences that say the same thing land at nearby points even with zero shared words.

"Cancel my subscription" and "I want to stop paying" end up close together. That's the whole trick: similar meaning becomes short distance, which is what makes semantic search possible.

A map of meaning

BEFORE: Old keyword search only finds exact words. Type "heart attack treatment" and it misses "myocardial infarction therapy" — same meaning, different words.

PAIN: Relevant results stay invisible unless the author used your exact vocabulary. You hit dead ends and never know what you missed.

MAPPING: Embeddings place text as coordinates on a map of meaning — "king" sits near "queen" like Paris sits near London. Cosine similarity measures the angle between two points: a small angle means similar meaning, whatever words were used.

Text to vector to match

1. An embedding model reads textA specialized network (not an LLM) outputs a fixed array of floats — typically 768–1,536 numbers.
2. Meaning shapes the numbersTrained on millions of pairs, it places similar meanings at nearby coordinates.
3. Compare by cosine similarityScore from 0 to 1; higher means closer meaning, regardless of length.
4. Search by nearnessEmbed the query, find the nearest stored vectors — those are your best matches.

Pseudocode

v1 = embed("heart attack treatment")
v2 = embed("myocardial infarction therapy")
v3 = embed("best pizza recipes")

cosine(v1, v2)   # ~0.95  same meaning
cosine(v1, v3)   # ~0.10  unrelated

# nearness in space = similar meaning
# use the SAME model to index + query

Misconceptions

Split docs into searchable pieces

Chunking splits documents into small segments before embedding. Why not embed the whole doc? Because embedding models work best on short text, and a 5,000-word doc becomes a blurry average of everything — hard to match to a specific question.

The whole game is granularity. Too big and one relevant sentence drowns in noise; too small and a chunk loses the context that made it meaningful. Chunking quality directly sets retrieval quality.

Making study flashcards

BEFORE: You have a 300-page textbook and must turn it into study flashcards — but you decide how much text goes on each card up front.

PAIN: Too big (a whole chapter per card) and you re-read pages to find one fact. Too small (one sentence per card) and "the treatment was effective" is meaningless — which treatment, for what?

MAPPING: Chunking is that exact trade-off. Each chunk is a flashcard the retriever hands the model, and the right card size depends on how dense and structured your material is.

Three strategies + overlap

1. Fixed-sizeSplit by character or token count. Simple and predictable, but may cut mid-sentence.
2. RecursiveSplit at natural boundaries first — headings, then paragraphs, then sentences.
3. SemanticDetect topic shifts by comparing embedding similarity and split where the topic changes.
4. Add overlapDuplicate 10–20% of text at each boundary so a sentence spanning two chunks isn't lost.

Which chunking to use

IF content is uniform / plain text:
  → fixed-size (~300-500 chars,
     10-20% overlap) — start here

IF docs have clear structure
   (headings, sections):
  → recursive (respect boundaries)

IF topics shift mid-document
   and quality still lags:
  → semantic (topic-shift splits)

# always tune on YOUR retrieval quality

Misconceptions

Storage built for similarity search

A vector database does one job well: store high-dimensional vectors and find the closest matches to a query vector, fast. It also keeps the original text and metadata beside each vector, so a search returns the actual chunk — not just numbers.

You never touch the raw vectors. You hand it a question, it embeds, searches, and returns ranked chunks with their source. ChromaDB, Pinecone, and pgvector are common choices.

A library shelved by topic

BEFORE: A traditional library shelves books alphabetically by author. To find everything on "machine learning in healthcare," you'd need to know every author and check each shelf.

PAIN: You miss relevant books because you didn't know the author, and you waste hours browsing shelves organized by something you don't care about.

MAPPING: A vector DB shelves books by what they're about. Everything on that topic sits together regardless of author, and you find it by describing the topic — not by knowing an exact title.

One row, then a query

1. Each row has four fieldsAn id, the vector, the original document text, and metadata (source, index).
2. The vector is what's searchedYour query gets embedded and compared by cosine similarity.
3. ANN skips the full scanApproximate nearest neighbor (HNSW) checks ~200 vectors, not 100,000 — O(log n).
4. Text + metadata come backYou get the top-K chunks to read, ranked by score, plus where they came from.

Pseudocode

# store: vector + text + metadata
db.add(
  id="chunk_14",
  vector=embed(text),
  document=text,
  metadata={ source: "guide.md" })

# query: DB embeds, ANN-searches, unpacks
hits = db.query("How long until lapse?", k=3)
FOR h IN hits:
  use(h.document, h.metadata, h.score)

Misconceptions

Never render a citation you haven't verified

Retrieval gives you chunks and an answer, but tying each claim back to its source is your job. Managed models ship a native citations API; open-source models on Ollama don't — there's no citations[] array.

The open-source pattern is just as audit-grade because you own the step that actually matters: verification. Force the model to cite a chunk id + a verbatim quote, then confirm the quote really appears in that chunk before showing it.

The fact-checker at a newspaper

BEFORE: A reporter files a story full of confident quotes. Before it prints, someone has to make sure each quote is real.

PAIN: Print the reporter's "[1] [2]" references as-is and a fabricated or misremembered quote sails through — and in legal or medical work, an unsourced claim is one you can't defend.

MAPPING: Your verification code is the fact-checker. The model proposes a quote and a source; your code checks the quote is truly a substring of that chunk. Only then does the citation count.

The citation contract

1. Tag every chunkGive each retrieved chunk a stable id (C1, C2…) in the context you send.
2. Demand structured citationsUse tool use to force the answer as claims, each with a chunk_id and a verbatim quote.
3. Verify, never trustCheck the id exists and the quote is genuinely a substring of that chunk's text.
4. Flag failuresAny claim that fails verification renders as UNVERIFIED — surfaced, not silently trusted.

Pseudocode

# force claims: {text, chunk_id, quote}
claims = model.answer_with_citations(
           question, tagged_chunks)

FOR claim IN claims:
  src = chunks[claim.chunk_id]
  claim.verified = src AND
    norm(claim.quote) IN norm(src)

# normalize case + whitespace first —
# models re-space quotes.
# show UNVERIFIED, don't hide it.

Misconceptions

Everything becomes text first

Real documents arrive as PDFs, scans, charts, and screenshots — not clean text. Hosted models can read a PDF natively; a text-only local model like Mistral can't. There's no "just send the PDF" shortcut.

So you convert everything to text upstream, then feed it into the exact RAG pipeline you already built. Digital PDFs get their text extracted, scans get OCR, and pixels-with-meaning go through a vision model that describes them.

The translator at the front desk

BEFORE: Imagine a brilliant analyst who only reads English, but documents arrive in French, as handwriting, and as photographs of whiteboards.

PAIN: Hand any of those straight to the analyst and they're stuck — they can't process a format they don't read, so the document is effectively invisible.

MAPPING: You put a translator at the front desk. Each format gets converted to plain English before it reaches the analyst: extractor for digital PDFs, OCR for scans, a vision model for pictures. The analyst only ever sees text.

Three conversion tracks

1. Digital PDFHas a text layer — pull prose and tables with pdfplumber or PyMuPDF.
2. Scanned pageJust an image of text — run OCR (Tesseract) at 300 DPI to recover it.
3. Chart / photoNo text to extract — a vision model (LLaVA, Llama Vision) describes it as text.
4. Converge to text → RAGAll three produce plain text that chunks, embeds, and retrieves unchanged.

Route each format

IF PDF has a text layer:
  → pdfplumber / PyMuPDF (near free)

IF scanned page or photo of text:
  → Tesseract OCR (rasterize @300 DPI)

IF chart, diagram, or photo:
  → vision model (LLaVA), cache result

IF unsure a PDF is scanned:
  → try extract_text(); empty? → OCR

Vision models are heavy — reserve them for genuine pixels-with-meaning and cache the text description.

Misconceptions

Test the core insight

One question per concept. Tap to reveal the answer.