M02 · The RAG Baseline and Where It Breaks

Learning objectives

RAG in two phases (a fast, honest recap)

You've likely built RAGRetrieval-Augmented Generation: retrieve relevant text from your own data and paste it into the model's prompt so it can answer from your knowledge instead of its training data. before (the sibling course spends two modules on it). Here is the whole machine in two phases, stripped to what matters for this module's argument.

Phase 1 — SETUP (done once, redone on every content change): take your documents, break them into small pieces — chunksFixed-size text segments, typically 256–1000 tokens, cut from documents so each piece fits an embedding model's input and stays "focused.", typically around 512 tokens. Convert each chunk into an embeddingA vector of hundreds of floats representing the text's meaning; texts with similar meaning get vectors that sit near each other.. Store the vectors in a vector database.

Phase 2 — QUERY (every single question): embed the user's question, find the nearest neighborsThe stored vectors closest to the query vector by cosine similarity — the mathematical stand-in for "most related text." among the stored chunk vectors, paste the winning chunks into the prompt, and let the model answer. That's it: search, then generate.

Notice what the machine fundamentally IS: a similarity engine. At no point does it know what any chunk means, whether a chunk is current, or how one chunk relates to another. It knows distances between vectors. Every failure in this module is that one sentence wearing different costumes.

Shredded context — what chunking destroys

The churn-rate incident — probabilistic retrieval meets canonical truth

Shredding destroys structure. The second failure is subtler: even perfectly intact chunks can be the WRONG chunks — and nothing tells you.

Here is the corpus's composite incident, assembled from real deployments, told in full because every beat matters. A data analyst asks the company's AI agent: "Write an executive SQL query calculating our churn rate for Q2." The agent embeds the query and searches a vector database holding thousands of chunked PDFs, Confluence pages, and archived Slack logs. It retrieves three high-similarity chunks: a PowerPoint deck from 2023, an old engineering wiki page, and — this detail is perfect — a Slack conversation of two data engineers arguing about how churn should be calculated.

All three chunks are genuinely "about churn." All three score high on similarity. And they contain three conflicting definitions. The model, given conflicting authorities and no way to rank them, blends. The query it writes pulls from the wrong schema. The dashboard ships. Nothing crashed. Nothing errored. Nobody notices for three weeks — until finance asks why weekly revenue doesn't match the invoice system.

The diagnosis, in the corpus's words: "the agent confidently retrieved the wrong version of the truth, because that's all a search engine can do when the underlying knowledge was never actually settled in the first place. You built a search engine. You needed a memory."

Generalize the incident and you get this module's central rule. Retrieval is probabilistic: it returns the most similar chunks, with no concept of authoritative. For exploratory questions ("what themes come up in churn discussions?") that's exactly right. For canonical facts — the metric definition, the compliance policy, the API contract, the SLA — similarity is the wrong criterion entirely. A canonical fact doesn't need to be rediscovered by search on every query. It needs to be written down once, correctly, and looked up. As the corpus puts it: for compliance docs, runbooks, and API specs, "90% semantic similarity is a failure; 100% exactness is non-negotiable."

The operational bill: drift, re-indexing, and the permissions leak

Two costs of vector pipelines rarely make it into the launch blog post.

Synchronization. Embeddings are a derived copy of your data, and every derived copy must be kept in sync with its source. Content changes mean re-chunking and re-embedding; embedding-model upgrades mean re-indexing everything (vectors from different models aren't comparable). The corpus's operators call keeping embeddings synchronized with fast-moving data "an absolute operational nightmare" — and stale vectors fail exactly like the churn incident: silently, by serving yesterday's truth with today's confidence.

The access-control leak. This one costs companies real incidents. Source systems — BigQuery, Cloud Storage, your wiki — enforce per-user permissions. A typical RAG deployment copies content out of those systems into a vector index… and the permissions usually don't come along. Result: anyone who can query the index can see everything in it, regardless of what they were allowed to see at the source. Finance-restricted tables, HR documents, incident postmortems — flattened into one searchable pool. This is among the most common, least-discussed ways retrieval systems quietly become security problems; it's specifically why Google built OKF's companion service (Knowledge Catalog) to keep access scoped to source-system permissions — the knowledge moves, and the access rules move with it.

Leads vs answers — the structural blind spot

Now aim RAG at code specifically, because the contrast with M01 becomes crisp. Semantic code search (what Cursor's built-in indexing does) is genuinely useful for exploration: "what files relate to authentication?" works well when you don't know where to look. But semantic search has a structural blind spot the corpus names precisely: it doesn't understand relationships. It doesn't know that handleAuth() calls validateToken(), which imports from jwt_utils. It knows these functions contain similar language.

The consequence shapes the agent's whole workflow. Semantic search returns leads — similarity-ranked hints the agent must then verify by opening files one by one (paying M00's exploration tax). A graph returns answers — definitive structural facts, pre-verified at extraction time. Same user question; fundamentally different information architecture. And recall M01's provenance lesson: the graph even labels which answers are facts vs guesses. A similarity score is not a truth label — a 0.91-similarity chunk can be flat wrong, and a 0.62 one exactly right.

Walk it, step by step

Watch the shredding happen. The retrieved chunks are all plausibly related to the question, and the answer built from them is confidently incomplete — which is the failure mode that costs you, not an empty result.

Where RAG still wins — and the professional default

If this module has made you want to delete your vector database: stop. That would be replacing one over-correction with another.

RAG was built to solve one problem well, and it still solves it better than anything in this course: finding relevant passages inside a huge pile of unstructured documents you cannot realistically curate ahead of time. Millions of support tickets. Years of Slack. A document lake of one-off design docs and PDFs. Nobody will ever write concept files for that corpus — and its questions are genuinely exploratory ("what themes come up in churn interviews?"), which is similarity's home turf.

Keep the base rates in view, too: industry surveys found roughly 72% of enterprise RAG deployments underdelivered or failed in year one — but the dominant causes were bad chunking strategies, stale indexes, and mismatched embedding models, not the absence of a graph. The corpus's warning deserves italics: adding a knowledge graph to a flawed retrieval pipeline gives you a flawed retrieval pipeline with higher operational overhead. And in high-stakes territory, retrieval alone measurably fails: legal-domain studies show RAG systems hallucinating on 17–33% of complex queries, because retrieval doesn't model multi-document logic.

So the professional default, per Atlan's guidance, is: "RAG first, graph when depth becomes a bottleneck." Run retrieval for the long tail. Watch for the two specific failure signatures — multi-hop structural questions failing (add a code graph: Track 2), canonical facts coming back stale or conflicting (add curated knowledge: Track 3). M10 turns this into a full decision matrix. Adding layers without an observed failure adds complexity, not capability.

Code walkthrough — make the failure happen, then fix it properly

Theory becomes conviction when the failure runs on your machine. orderflow's docs/architecture.md deliberately preserves a stale paragraph — an old WAU definition ("any API call in 7 days," no tester exclusion) kept under a "Historical note" heading, exactly like real wikis keep them. The canonical definition lives in services/orders/metrics.py. We'll build a tiny retrieval pipeline that surfaces the stale one, then answer the same question deterministically.

Chunk 1 — the chunker. We split docs on paragraph boundaries into ~400-char pieces, keeping {text, source, chunk_index}. Note what we're already losing at this line: the "Historical note (STALE)" heading that qualifies the paragraph can land in a different chunk than the definition it disclaims. The shredder, in one function. Gotcha: we use keyword overlap instead of real embeddings — deliberately. The failure we're demonstrating is about what similarity ranks, not how similarity is computed; keyword scoring shows it without an API key, and a real embedder makes it worse, not better, because the stale paragraph is semantically immaculate.

Chunk 2 — retrieval. Score every chunk by query-term overlap, return the top 3. The stale paragraph is keyword-dense ("weekly active users… 7 days…") so it ranks — often above the correct source, which phrases things differently in code.

Chunk 3 — the deterministic alternative. Same question, zero search: read ONE known file (the M06-style concept for WAU) and print its Computation Rules. No ranking, no luck, no conflicting authorities — a lookup, because a canonical fact deserves a lookup.

from pathlib import Path

DOCS = Path("../sample-project/docs")

# ---- Chunk 1: the shredder ----
def chunk_docs(max_len=400):
    chunks = []
    for path in sorted(DOCS.glob("*.md")):
        try:
            text = path.read_text(encoding="utf-8")
        except OSError as exc:
            print(f"skip {path}: {exc}")      # never let one bad file kill the pipeline
            continue
        buf = ""
        for para in text.split("\n\n"):
            if len(buf) + len(para) > max_len and buf:
                chunks.append({"text": buf, "source": path.name, "i": len(chunks)})
                buf = ""
            buf += para + "\n\n"
        if buf.strip():
            chunks.append({"text": buf, "source": path.name, "i": len(chunks)})
    return chunks

# ---- Chunk 2: similarity by keyword overlap (embeddings make this WORSE, not better) ----
def retrieve(query, chunks, k=3):
    terms = set(query.lower().split())
    scored = sorted(chunks,
                    key=lambda c: -len(terms & set(c["text"].lower().split())))
    return scored[:k]

chunks = chunk_docs()
print(f"{len(chunks)} chunks indexed")
for c in retrieve("how is weekly active users calculated?", chunks):
    print(f"--- {c['source']} #{c['i']} ---\n{c['text'][:180]}...\n")
# One of these is the STALE definition from architecture.md's historical note.
# Nothing crashed. The pipeline 'worked'.

# ---- Chunk 3: the deterministic alternative — a lookup, not a search ----
import frontmatter
concept = frontmatter.load(
    "../M06-okf-authoring/solution/knowledge/analytics/metrics/weekly_active_users.md")
print(concept["title"])          # exactly one authority,
print(concept.content)           # version-controlled, current, whole.

import { readFileSync, readdirSync } from "node:fs";
import { join } from "node:path";

const DOCS = "../sample-project/docs";

// ---- Chunk 1: the shredder ----
function chunkDocs(maxLen = 400) {
  const chunks = [];
  for (const name of readdirSync(DOCS).filter((f) => f.endsWith(".md")).sort()) {
    let text;
    try { text = readFileSync(join(DOCS, name), "utf-8"); }
    catch (err) { console.error(`skip ${name}: ${err.message}`); continue; }
    let buf = "";
    for (const para of text.split("\n\n")) {
      if (buf.length + para.length > maxLen && buf) {
        chunks.push({ text: buf, source: name, i: chunks.length });
        buf = "";
      }
      buf += para + "\n\n";
    }
    if (buf.trim()) chunks.push({ text: buf, source: name, i: chunks.length });
  }
  return chunks;
}

// ---- Chunk 2: similarity by keyword overlap ----
function retrieve(query, chunks, k = 3) {
  const terms = new Set(query.toLowerCase().split(/\s+/));
  return [...chunks]
    .map((c) => ({ c, score: c.text.toLowerCase().split(/\s+/).filter((w) => terms.has(w)).length }))
    .sort((a, b) => b.score - a.score)
    .slice(0, k).map((x) => x.c);
}

const chunks = chunkDocs();
console.log(`${chunks.length} chunks indexed`);
for (const c of retrieve("how is weekly active users calculated?", chunks)) {
  console.log(`--- ${c.source} #${c.i} ---\n${c.text.slice(0, 180)}...\n`);
}
// One of these is the STALE definition. Nothing crashed. The pipeline "worked".

// ---- Chunk 3: the deterministic alternative ----
const concept = readFileSync(
  "../M06-okf-authoring/solution/knowledge/analytics/metrics/weekly_active_users.md", "utf-8");
console.log(concept);   // one authority, version-controlled, current, whole

Hands-on lab — labs/M02-rag-shreds-context

📂 Get the files: labs/M02-rag-shreds-context on GitHub — or clone the whole course once: git clone https://github.com/varasrinivas/knowledge-graph-course.git

What you'll build: the pipeline above, from scratch, plus a short written reflection. Time: 25–35 min. Prerequisites: M01 lab; pip install python-frontmatter.

1. Step 1 — Chunk the docs. Write chunker.py splitting sample-project/docs/ into ~400-char paragraph-aligned chunks. Run it. ✅ Expect ~8–15 chunks with {text, source, chunk_index}.
2. Step 2 — Retrieve. Write retrieve.py scoring by term overlap for "how is weekly active users calculated?"; print the top 3. ✅ The stale architecture.md paragraph appears among them.
3. Step 3 — The deterministic alternative. Load the M06 solution's WAU concept with python-frontmatter; print its Computation Rules. ✅ 7-day window, completed orders, tester exclusions.
4. Step 4 — Reflect (in comments). (a) Would a better embedding model fix Step 2? (b) What process would have prevented the stale paragraph from existing? You'll build exactly that process in M08.

Troubleshooting: if the stale paragraph doesn't surface, your chunker may be merging whole files into one chunk — check the max-length split fires; the demonstration depends on the disclaimer heading separating from the definition.

Knowledge check

Module summary

Next — M03: Parsing Code into Graphs with tree-sitter. Track 2 begins: where M01's edges actually come from. You'll parse real orderflow source into an AST, watch a fault-tolerant parser shrug off syntax errors, and extract call edges with zero LLM calls — the deterministic pass that cannot hallucinate.

References

• Prism Labs — Beyond Vector Embeddings in 2026 (the 512-token co-location analysis)
• Atlan — Knowledge Graphs vs RAG for AI (2026): "RAG first, graph when depth becomes a bottleneck"
• Google Cloud — Knowledge Catalog (source-permission-preserving serving for OKF bundles)
• Legal-domain RAG hallucination studies (17–33% on complex multi-document queries)
• The stale-definition fixture: labs/sample-project/docs/architecture.md (Historical note)
• Lab: labs/M02-rag-shreds-context · canonical WAU: labs/M06-okf-authoring/solution/knowledge/analytics/metrics/weekly_active_users.md