Knowledge Graphs for AI Agents · Capstone

Capstone — Build a Self-Updating Knowledge Graph Pipeline

Module 14 of 14 — the final build
⏱ ~2–3 hours Difficulty: ★★★★☆ (12–18 steps) 🔗 Prerequisites: M01–M12 — this module reuses your lab solutions from M01, M06/M07, M08, M09, and M11

Project Brief

You are the platform engineer for orderflow, the B2B order-tracking system you have been graphing all course. Your team runs several AI agents against the repo daily, and two chronic problems finally boiled over. First, the token bill: every agent session starts with the same archaeology — grep for callers, open files, trace imports — re-deriving structure that has not changed in weeks. Second, and worse: last month an agent answered a revenue question using the stale WAU definition from an old paragraph in docs/architecture.md — "any API call in 7 days, no tester exclusion" — instead of the canonical computation in services/orders/metrics.py. Nothing crashed. The number was confidently, quietly wrong, and finance caught it three weeks later.

Your job in this capstone is to build the complete fix — the full three-layer context system this course taught, with the maintenance machinery that keeps it honest:

  • A structural layer: a deterministic code graph (Track 2)
  • A narrative layer: a curated OKF bundle with a canonical WAU concept (Track 3)
  • A serving layer: an MCP server answering structural questions in one tool call (Track 4)
  • A self-updating pipeline: a diff-scoped enrichment hook (Track 3)
  • A freshness gate: CI that catches a stale graph within one commit (Track 5)

This is a BUILD module: roughly 80% lab, 20% concept. Everything you need already exists in your labs/ folder — the point is wiring your own M01, M06/M07, M08, M09, and M11 solutions into one system, then sabotaging it deliberately and proving your defenses catch the failure. The full lab packet mirrors this page at labs/CAPSTONE-knowledge-pipeline/README.md.

Before modern aviation, a pilot's "preflight check" was looking at the plane and deciding it seemed fine — and planes crashed for reasons a checklist would have caught. The pain: every individual system could work while the aircraft as a whole was unairworthy, because nobody verified the connections between systems. The mapping: you have built five working components across ten labs. A capstone is the airworthiness test — not "does each part run" but "does the hook actually feed the graph the server actually serves, and does the gate actually block when one link silently fails?" The sabotage steps below are your checklist items; a system you have never watched fail is a demo, not infrastructure.

Architecture — What You're Assembling

The finished system is a context pipeline: every git commit triggers a diff-scoped refresh of two artifacts — the structural graph.json (extracted, zero-LLM) and the narrative knowledge/ bundle (concept files with cross-links). An MCP serverA Model Context Protocol server — a small process exposing typed tools (here: graph queries) to any MCP-aware agent over stdio. serves both to agents. A CI freshness gate compares artifact timestamps against git log -1 and blocks any pipeline that would let an agent trust a stale map.

✅ Why this shape and not another

Each layer earns its place by a failure mode you observed in the course: multi-hop questions failing → structural graph (M01/M04). Tribal knowledge and stale definitions → curated narrative concepts (M06–M08). Exploration burning the context window → one-call MCP serving (M09). Silent drift → the gate (M11). Remove any one and a specific, named failure returns. That is the M10 discipline: layers justified by failure modes, never by fashion.

File Structure

where everything lands (inside labs/sample-project/)
sample-project/                      # your working repo (git init'd in Phase 4)
  graphify-out/
    graph.json                       # structural layer — regenerated by the hook
  knowledge/                         # narrative layer — refreshed by the hook
    index.md                         # the agent's pre-flight read
    log.md                           # what the bundle KNOWS changed (≠ git log)
    services/                        # billing / orders / notifications concepts
    analytics/                       # orders_fact + weekly_active_users concepts
  .git/hooks/post-commit             # M08 hook — diff-scoped, loud on failure
  ci/
    freshness_gate.py                # M11 gate (copied in, Phase 6)
    ci.sh                            # gate + lint, exit 1 on either failure
  BASELINE.md                        # your before/after measurements
  NOTES.md                           # god node analysis + the honest writeup

The Build — Seven Phases, Eighteen Steps

📂 Get the files: labs/CAPSTONE-knowledge-pipeline on GitHub — or clone the whole course once: git clone https://github.com/varasrinivas/knowledge-graph-course.git

Phase 1 — Baseline Measurement (steps 1–2)

You cannot claim an improvement you never measured. Before any tooling, establish what agent work costs on the bare repo.

Step 1

Ask the three questions, cold

What & why: with NO graph, no bundle, no MCP server registered, ask your coding agent these three questions and record, for each, the number of files it reads and (if your client shows it) tokens consumed:

  • Q1 — "Who calls decode_jwt, directly and transitively?"
  • Q2 — "What breaks if orders_fact gains a column?"
  • Q3 — "What is the canonical weekly-active-users definition?"

Watch Q3 especially: there is a deliberately stale definition in docs/architecture.md, and a cold agent may well retrieve it — reproducing the incident from the brief on your own machine.

Step 2

Record the numbers

What & why: create BASELINE.md with one row per question: files read, tokens (or your best proxy), and whether the answer was correct. This table gets its "after" columns in Phase 5.

✅ Checkpoint

Three rows, each with reads + tokens + a correctness verdict. If Q3 came back with the stale definition, note it — that row is the whole business case.

Phase 2 — The Structural Layer (steps 3–5)

Step 3

Generate the graph

What & why: the structural layer is deterministic extraction — zero LLM calls, zero hallucination risk in the structural pass (M03). Use Graphify if installed, or the course's pure-Python extractor:

bash — either path works
# with Graphify installed (M04):
cd labs/sample-project
graphify update .

# offline fallback — same repo, same edges:
python ../shared_tools/kg_extract.py . --out graphify-out/graph.json
Rebuilt: 36 nodes, 22 edges -> graphify-out/graph.json nodes: {'class': 4, 'function': 18, 'module': 14} edges: {'EXTRACTED': 9, 'INFERRED': 13}

Troubleshooting: graphify: command not found → uv/pipx bin dir not on PATH; use the fallback and continue — every later phase works with either.

Step 4

Verify against ground truth

What & why: orderflow is deliberately small enough to check by hand — that is why the course uses it. Open sample-project/README.md and confirm every listed relationship appears in your graph: verify_token → decode_jwt (EXTRACTED), webhooks → mark_paid + publish, worker → subscribe, orders → DatabasePool.execute.

✅ Checkpoint

All ground-truth edges present; provenance tags make sense (same-module resolutions EXTRACTED, cross-module bare-name matches INFERRED). If an edge is missing, your extractor skipped a file — check for syntax errors in your local copies.

Step 5

Find the communities and the god node

What & why: load graph.json (or reuse your M01 solution) and compute in-degree per module. Then write one paragraph in NOTES.md: what does it mean for refactoring risk that one module tops the list?

python — quick in-degree tally
import json
from collections import Counter
g = json.load(open("graphify-out/graph.json"))
indeg = Counter(e["target"].split(":")[1].rsplit(".", 1)[0] for e in g["edges"])
print(indeg.most_common(3))
✅ Checkpoint

shared.auth tops the list at 6, with shared.db just behind at 5 — the shared cluster the architecture doc warned about. Note that this is in-degree — one measure among several, and M01 showed they disagree: rank orderflow by betweenness and shared/db.py scores 0.00 (it is a sink), while rank by blast radius and it wins outright with seven transitive dependents. Your paragraph should say which measure you chose and why, then connect it to blast radius: a change in shared/db.py can alter behaviour in seven functions across billing and orders, so changes there need the widest testing. (Notifications is not among them — it consumes events and never persists.)

Phase 3 — The Narrative Layer (steps 6–8)

The graph knows what the code IS. It cannot know that WAU excludes testers on purpose, or that the event bus was chosen over a broker in ADR-001. That intent lives in the bundle — what engineers KNOW, written once, curated (M06/M07).

Step 6

Complete the orderflow bundle

What & why: finish your M07 bundle: knowledge/index.md, log.md, three Service concepts, plus the two analytics concepts from M06 (orders_fact, weekly_active_users). Keep the body structure identical across Service concepts — # Responsibilities then # Dependencies — consistency is what agents parse (M07).

Step 7

Lint until clean

bash
python ../M06-okf-authoring/solution/validate.py knowledge
5 concepts valid, 0 errors

Troubleshooting: YAML errors → check the --- delimiters sit alone on their lines; missing-type errors name the exact file.

Step 8

Make WAU an attested computation

What & why: the spec's answer to the stale-definition incident is the attested computationAn OKF concept documenting the sanctioned, checkable WAY to compute a value — not just what the value means. The single source of truth for governed metrics.: the concept must point at the code that IS the definition. Ensure weekly_active_users.md names services/orders/metrics.py::weekly_active_users as the source of truth and states the exclusion rule explicitly.

✅ Checkpoint

Lint passes; every concept cross-links in BOTH directions (billing → notifications for publishes, notifications → billing for consumes). One-directional links are how bundles silently rot (M07).

Phase 4 — Self-Updating (steps 9–11)

A bundle nobody maintains is a wiki nobody reads — and a graph nobody rebuilds is a map that lies (M08, M11). This phase installs the machinery, then proves it fails LOUDLY, because the failure mode that kills these systems is the silent one.

Step 9

Install the hook

bash
cd labs/sample-project
git init -q && git add -A && git commit -qm "baseline"
cp ../M08-enrichment-hook/solution/post-commit .git/hooks/post-commit
# macOS/Linux: chmod +x .git/hooks/post-commit

Note what this hook deliberately does NOT do: no nohup (silently absent on Git-for-Windows — failure mechanism #1 from M11), no backgrounding, no silent resource-gated skips. Every failure prints and exits non-zero.

Step 10

Trigger a real refresh

What & why: simulate the M08 walkthrough — a new dependency appearing in billing. Add a comment (or a fake fraud_check() call) to services/billing/webhooks.py and commit.

[enrich] changed: services/billing/webhooks.py -> concept billing-service [enrich] refreshed Dependencies (4 edges) in billing-service.md [enrich] log.md updated [enrich] lint: 5 concepts valid, 0 errors
✅ Checkpoint

The diff-scope worked: ONLY billing-service.md was refreshed (orders and notifications untouched), and log.md gained a dated entry. Diff-scoping is the entire cost model — full-repo rescans per commit are what make teams turn pipelines off (M08).

Step 11

Sabotage #1 — break the lint gate

What & why: delete the type: line from any concept, then commit any code change. The hook must FAIL, non-zero, naming the file:

ERROR knowledge/services/billing-service.md: missing required field 'type' 1 concepts valid, 1 errors [enrich] FAILED — bundle NOT updated. Fix before trusting the graph.
✅ Checkpoint

Loud failure observed, then fixed. If the hook passed silently, your lint gate is decorative — exactly the "technically conformant, practically unreliable" trap the spec's leniency creates (M06/M11). Restore the type: field before continuing.

Phase 5 — Serving (steps 12–13)

Step 12

Register the MCP server

bash — selftest first, then register
cd ../M09-mcp-graph-server/solution
python server.py --selftest        # must print SELFTEST PASSED
claude mcp add orderflow-graph -- python /absolute/path/to/labs/M09-mcp-graph-server/solution/server.py

Troubleshooting: tool calls hang → something printed to stdout; stdio MCP servers must log to stderr only (M09).

Step 13

Re-ask the three questions

What & why: repeat Q1–Q3 with the server registered and the bundle in place; append the "after" columns to BASELINE.md.

✅ Checkpoint

Q1 and Q2 each answered in ONE tool call (graph_callers / explore — blast radius included). Q3 answered from ONE concept file, with the canonical tester-exclusion rule, because index.md routed the agent straight to the attested computation. The stale doc never entered context.

Phase 6 — Monitoring (steps 14–16)

Step 14

Install and pass the gate

bash
mkdir -p ci && cp ../M11-freshness-gate/solution/freshness_gate.py ci/
python ci/freshness_gate.py . --max-age-commits 0 --cross-check
FRESH: graph.json generated at/after HEAD commit CROSS-CHECK OK: stored counts match a fresh extraction
Step 15

Sabotage #2 — commit without rebuilding

bash — bypass the hook deliberately
git commit --allow-empty -qm "commit without rebuilding the graph" --no-verify
python ci/freshness_gate.py . --max-age-commits 0
STALE: graph.json predates HEAD by 1 commit(s) fix: python ../shared_tools/kg_extract.py . --out graphify-out/graph.json (then restart any MCP server holding the old graph)
✅ Checkpoint

Exit 1, an actionable message, AND the reminder about the MCP startup cache — the fourth silent-failure mechanism from M11: a fresh graph on disk is still stale in a running server's memory until you bounce it. Fix by re-running the extractor and restarting the server.

Step 16

Wire ci.sh

ci/ci.sh — the whole defense in four lines
#!/bin/sh
set -e
python ci/freshness_gate.py . --max-age-commits 0 --cross-check
python ../M06-okf-authoring/solution/validate.py knowledge
✅ Checkpoint

sh ci/ci.sh exits 0 on the healthy repo, non-zero if EITHER the graph is stale or the bundle fails lint. This script is your answer to the 24-hour question — run it on a schedule and staleness cannot hide for more than one interval.

Phase 7 — The Honest Writeup (steps 17–18)

Step 17

Compute your real multiple — and explain why it's small

What & why: divide your Phase-1 token counts by Phase-5's. On a 15-file repo, expect a single-digit multiple — and write down in NOTES.md why that is the CORRECT, expected result, not a disappointment. orderflow sits far below the ~500-file threshold where graph tooling pays for itself (M04, M11); the exploration a graph eliminates barely exists at this scale. The 71.5x headline was a ceiling case on a favorable 52-file corpus; independent replications landed at 6.8x–49x, 7.3x on a from-scratch real Python codebase, and savings scale with the size of the haystack — VS Code's 10,000 files saved 78%, OkHttp's 645 files saved 13% (M04/M05). Your small number on a tiny repo is the benchmark-literacy lesson, experienced firsthand.

Step 18

Answer the defining question

What & why: in NOTES.md, answer in specific mechanical terms: "How would I detect a stale graph within 24 hours?" Your answer should name the gate script, what it compares (artifact mtime + counts vs git log -1), where it runs (CI + a scheduled catch-up), what it does on failure (exit 1, actionable message, alert), and the MCP restart step. If you can write that paragraph, you have a production setup. If you cannot, you have a demo that hasn't failed yet — the M11 test, applied to your own work.

🎉 Final checkpoint

BASELINE.md has before/after rows; NOTES.md has the god-node paragraph, the honest multiple, and the 24-hour answer; both sabotages were caught loudly; ci.sh guards the repo. The system is airworthy. Course complete.

Walk it, step by step

The whole pipeline in one pass, one stage per step, with the artifact each stage produces and the check that proves it worked. Use it as the map while you build.

Test Cases

Run all five. Three happy paths prove the layers work; the edge and failure cases prove the system degrades honestly — which this course has argued is the more important property.

#TypeInputExpected
1happyMCP graph_callers("decode_jwt")["shared.auth.verify_token"] — one call, no file reads
2happyMCP explore("execute")callers across billing AND orders; blast radius > 3 — the god node's reach, quantified
3happy"What is the canonical WAU definition?" via the bundle7-day window, completed orders, tester exclusion — from the concept file, citing metrics.py; the stale doc never surfaces
4edgeMCP graph_callers("stripe_sdk_call")graceful error: … not in graph — try grep; the agent falls back to search, no crash. A graph that pretends to know external symbols is worse than one that says "not mine"
5failurecommit with --no-verify, then sh ci/ci.shfreshness gate exits 1 naming the lagging artifact and the fix — the silent failure made loud

Going Further (all OPTIONAL)

  • Two-pass enrichment — add M08's citation pass: after refreshing a concept, cross-reference ADR-001 and the runbook, linking each dependency to the document that explains WHY it exists. Skipping the citation pass is how plausible-but-unverified descriptions creep in.
  • Serve the bundle too — add a concept(path) MCP tool returning a concept file's body, so the narrative layer is one tool call away like the structural one.
  • Build the M10 router — classify incoming questions (canonical / structural / exploratory) and dispatch to bundle read, graph query, or grep automatically.
  • Comparison run — run okf-rs or CodeGraph over orderflow and diff their edges against kg_extract's using the M05 comparison axes.
  • Export to Obsidian — render the bundle as a linked vault and walk your knowledge graph visually.

Final Knowledge Check — the Whole Course

Seven questions spanning all six tracks. This is the exam-shaped recap.

1. (Track 1) Why did the cold agent in Phase 1 risk answering Q3 with the stale WAU definition?

The model was too small to understand the metric
Similarity-based retrieval surfaces keyword-dense text regardless of canonicity — probabilistic search is the wrong criterion for a canonical fact
The stale paragraph was longer than the correct code
Markdown files rank higher than Python files in search

2. (Track 2) The structural pass of your extractor made zero LLM calls. What does that buy you?

Faster answers, but with some hallucination risk
Determinism: identical source → identical graph, no API cost, and zero hallucination risk in extracted edges — which is why provenance tags can honestly say EXTRACTED
Better semantic understanding of what the code means
Automatic documentation of business intent

3. (Track 3) Why must the enrichment hook scope its scan to the git diff?

Full scans produce incorrect edges
The OKF spec forbids full-repository scans
Cost: a full-repo rescan per commit makes the pipeline too expensive to keep running — diff-scoping is what makes per-commit updates affordable
git hooks cannot see unchanged files

4. (Track 4) Your MCP server's explore tool bundles callers + callees + blast radius into one call. What's the design rationale?

MCP limits servers to a small number of tools
Tool-call overhead is context too — one composite answer keeps the agent's token budget for reasoning instead of a chain of round-trips
Composite answers are easier to cache on disk
Separate tools would require separate graph files

5. (Track 5) In Step 15 the gate caught a stale graph — but M11 warns one more thing can STILL be stale after you regenerate. What?

The git index
The bundle's log.md
A running MCP server's in-memory copy — it loaded graph.json at startup and has no hot-reload; restart it or the agent still queries the old graph
The .graphifyignore file

6. (Honesty rule) Your Phase-7 multiple came out at 3x, not 71.5x. The correct interpretation is:

The pipeline is broken; a correct setup always yields 70x+
Expected: token savings scale with repo size, orderflow is far below the ~500-file payoff floor, and the 71.5x figure was a ceiling case on a favorable corpus — replications landed at 6.8x–49x
The extractor missed most of the edges
MCP overhead consumed the savings

7. (The defining question) What makes a knowledge-graph setup "production-grade" per this course?

Benchmark numbers above 50x token reduction
Using all three layers plus a vector index
Its failure-detection story: a specific, mechanical answer to "how would I detect a stale graph within 24 hours" — gates, cross-checks, alerts, and server restarts you have watched fire
Automatic nightly full-repo rescans

Completion — What You Built, and What You Now Know

The systemStructural graph + OKF bundle + diff-scoped hook + MCP serving + CI freshness gate — every layer justified by a failure mode you reproduced.
The measurementsA BASELINE.md with real before/after numbers, and the literacy to explain why they're small on a small repo.
The sabotagesA lint gate and a freshness gate you watched FAIL loudly — the difference between infrastructure and demo.
The writeupA mechanical answer to the 24-hour staleness question, in your own words, about your own system.

The course's whole argument, compressed: agents don't need smarter models to stop wasting tokens — they need maps. Deterministic maps of what the code is, curated maps of what engineers know, served one tool call at a time, and monitored like the production infrastructure they are. You have now built each layer by hand, connected them, broken them, and caught the breakage. Whatever tools this fast-moving ecosystem ships next, that skill set — and the skepticism that comes with it — transfers.

🎓 Congratulations — you have completed Knowledge Graphs for AI Agents.

References

  • Lab packet: labs/CAPSTONE-knowledge-pipeline/README.md — and the reused solutions in labs/M01, M06–M09, M11
  • Graphify (github.com/safishamsi/graphify) · Google OKF spec (GoogleCloudPlatform/knowledge-catalog) · okf CLI (superops-team/okf) · okf-rs (jyjeanne/okf-rs) · CodeGraph
  • Course honesty sources: Graphify BENCHMARKS.md; independent replications (6.8x–49x; 7.3x from-scratch); the silent-hook postmortems and the ERPNext hybrid case study (70.8% → 82.0% key-fact coverage)
← PreviousM12 — Structuring Projects + What's Next