M08 — The Enrichment Pipeline: Self-Updating Graphs

Learning Objectives

• Explain why bundle maintenance — not the format — is the real engineering work of adopting OKF.
• Describe Google's two-pass enrichment pattern (draft, then cite) and what breaks when the citation pass is skipped.
• Assemble the six-stage pipeline: diff-scoped scan → draft/update → re-link → lint → publish, triggered by a git hook.
• Argue, with numbers, why diff-scoping is the difference between a pipeline that runs on every commit and one that gets disabled.
• Implement a working, loudly-failing enrichment hook for orderflow.

The Gap the Spec Deliberately Leaves

Here is the central claim of this module, and arguably of the whole Track: adopting OKF is trivial — one required field, a folder convention, plain Markdown links. The real engineering work is the enrichment pipeline that keeps the graph accurate and current as the code changes. The spec has no opinion on how a bundle gets generated or stays true. That is left entirely — and intentionally — to whoever adopts it.

Why does this gap matter so much? Because a knowledge graph that lags reality doesn't degrade gracefully — it lies confidently. An agent reading a bundle that says billing-service has two dependencies, when a third was added last sprint, will happily plan a refactor that breaks the third. Stale context isn't just slow; it's actively wrong, and wrong with the full authority of "documented knowledge." (Module 11 is an entire postmortem on how this happens silently even when you think you've automated it.)

The reason to be optimistic anyway comes from the LLM Wiki insight the whole ecosystem is built on: "LLMs don't get bored, don't forget to update a cross-reference, and can touch 15 files in one pass. The bookkeeping that causes humans to abandon personal wikis is exactly what LLMs are good at." Humans let documentation rot because upkeep is tedious. The enrichment pipeline's bet is that tedium is precisely the workload to delegate to a background agent — with mechanical guardrails around it.

The Two-Pass Agent: Draft, Then Cite

Google's own reference implementation — the BigQuery metadata agent — established the pattern everyone now copies. It is a two-pass agent:

Translated to code, pass 1 walks every changed module/service/API in the repo and drafts a concept from its interfaces and call graph; pass 2 adds citations back to the runbooks, ADRs, and PRs that explain why. The passes have different jobs and different failure modes. Pass 1 without pass 2 produces text that is plausible-sounding but unverified — the LLM describes what the code seems to do, with no anchor to human-recorded intent. That's precisely the "confident but wrong" content you built this system to eliminate. Keep the two-pass structure; a single-pass pipeline is a rumor mill with good formatting.

The Pipeline, Stage by Stage

Wrap the two-pass agent in triggering, scoping, and gating, and you get the pipeline every serious deployment converges on:

commit pushed
      |
      v
diff-scoped module scan        (which services/files actually changed?)
      |
      v
draft / update concept docs    (two-pass enrichment agent)
      |
      v
re-link cross-references       (update Dependencies AND Dependents, both sides)
      |
      v
lint                           (spec-compliance rules — a HARD gate)
      |
      v
publish                        (commit the bundle / push through CI / register to a catalog)

Two stages deserve emphasis beyond the diagram. Re-link runs on both sides: when billing-service gains a dependency on fraud-check, billing's # Dependencies gains a link and fraud-check's # Dependents (if it exists) gains the back-link. One-directional updates are how the M07 warning — bundles silently rotting — actually happens in practice. Lint is a hard gate, not a report: if the enrichment agent produced structurally invalid output, the pipeline must refuse to publish. A lint that only warns is decoration; the whole trust model of the bundle rests on "what's published passed the gate."

The pipeline also maintains log.md as it goes — every refresh appends a dated entry recording what the bundle learned. That's how the M06 distinction (knowledge history vs file history) gets kept true mechanically rather than by heroic discipline.

Diff-Scoping: The Entire Cost Model in One Decision

If you remember one sentence from this module: scope every enrichment pass to the git diff, not the whole repository. The entire cost model of self-updating knowledge hangs on this choice.

Run the numbers on a mid-size repo. A full-repo enrichment pass over 400 services means 400 concept drafts through an LLM — call it hundreds of thousands of tokens — per commit, at forty commits a day. That pipeline gets disabled by Friday of its first expensive week, and the bundle starts rotting the moment it's off. A diff-scoped pass on a typical commit touches one or two services: two drafts, two citation lookups, a lint run. That's cheap enough to run on every push, which is the only cadence that keeps the graph honest. As the field puts it: a team that re-scans everything on every commit will find the pipeline too expensive to keep running well before their repo gets large enough to actually need it.

End-to-End Trace: One Commit, Both Sides, Next Day

Walk the whole thing with concrete files. A developer adds a fraud-check call to orderflow's billing webhooks:

1. A developer pushes a change to services/billing/webhooks.py, adding a dependency on a new fraud-check service.
2. The git hook fires and triggers a diff-scoped scan — only billing-service and anything directly touched by the diff gets re-examined. Not the whole repository.
3. Pass 1 drafts the update: reading the service's new interface and call sites, it rewrites the # Dependencies section of billing-service.md to include fraud-check.
4. Pass 2 adds citations: it cross-references the change against the PR description explaining why the dependency was added, and links it in.
5. Cross-references update on both sides — billing-service.md now links to the fraud-check concept, and fraud-check.md's # Dependents section gains the back-link.
6. Lint runs against the updated bundle, catching anything the enrichment pass got structurally wrong.
7. The bundle publishes — committed alongside the code change, or pushed through CI to a served location.
8. The next day, an orchestrator picks up an unrelated task touching fraud-check. It reads index.md, loads just that one concept — including its now-current link back to billing-service — and never re-scans either codebase to understand the relationship.

Walk it, step by step

One commit, walked through the hook end to end. Watch for the failure mode this module keeps warning about: the hook that runs, exits zero, and updates nothing.

Bridging Structural → Narrative: graphify export

You don't have to draft pass-1 content from thin air. On July 1, 2026 — three weeks after the OKF spec shipped — a bidirectional Graphify↔OKF integration toolkit landed, exporting Graphify's structural graph straight into an OKF-compliant markdown catalog:

graphify export --format okf --out docs/knowledge/

The pitch is elegant: Graphify gives you precise "what calls what" (deterministic, zero-LLM, from Track 2); OKF gives you the portable, agent-friendly "why this exists." Structural graph plus narrative catalog, one pipeline. Bundles conventionally live at docs/knowledge/ or .well-known/okf/ in the repo. This is exactly how the labs' enrichment hook works: the structural graph supplies the Dependencies facts mechanically, and the LLM (when used at all) only writes the connective prose — the division of labor that keeps hallucination out of the load-bearing parts.

When — and When Not — to Build This

The honest decision framework, because pipelines have carrying costs:

Build it if you already run multiple agents against your repo or knowledge base. The pipeline pays for itself directly: fewer tokens re-deriving context per task, fewer stale-context bugs from agents acting on outdated mental models. The minimal viable version fits in an afternoon: okf init, a git hook, one enrichment pass over your most-changed service — not the whole repo — and index.md as the entry point every agent reads first. Measure the token delta on your next multi-agent task, and decide from data.

Skip it — for now — if you're not running agentic workflows against your code. OKF's entire value proposition is agent consumption. Without agents consuming it, "you are maintaining a wiki nobody reads." That's not a mild inefficiency; it's the exact failure mode (docs rot because upkeep has no consumer) this whole system was invented to escape.

Code Walkthrough: A Working, Loudly-Failing Hook

Let's build the orderflow enrichment hook — the same one shipped in labs/M08-enrichment-hook/solution/. It is deliberately LLM-free: pass 1 derives Dependencies from the structural graph, which makes the lab runnable offline and demonstrates that the pipeline's skeleton is deterministic engineering, not prompt magic.

Chunk 1 — find what changed. WHAT: ask git for the files in the last commit and map them to owning services. WHY: this is the diff-scope filter — everything downstream only runs for touched services. GOTCHA: normalize path separators; on Windows, git can hand you backslashes that break prefix matching.

import json, subprocess, sys
from datetime import datetime, timezone
from pathlib import Path

SERVICE_MAP = {                                # changed-path prefix -> (concept, module prefix)
    "services/billing": ("services/billing-service.md", "services.billing"),
    "services/orders": ("services/orders-service.md", "services.orders"),
    "shared": ("services/shared-libraries.md", "shared"),
}

def changed_files(repo: Path) -> list[str]:
    result = subprocess.run(["git", "diff", "--name-only", "HEAD~1", "HEAD"],
                            cwd=repo, capture_output=True, text=True)
    if result.returncode != 0:                # first commit, detached head, etc. — fail LOUDLY
        raise RuntimeError(f"git diff failed: {result.stderr.strip()}")
    return [ln.strip().replace("\\", "/") for ln in result.stdout.splitlines() if ln.strip()]

import { execFileSync } from "node:child_process";
import fs from "node:fs";
import path from "node:path";

const SERVICE_MAP = {                          // changed-path prefix -> [concept, module prefix]
  "services/billing": ["services/billing-service.md", "services.billing"],
  "services/orders": ["services/orders-service.md", "services.orders"],
  "shared": ["services/shared-libraries.md", "shared"],
};

function changedFiles(repo) {
  try {
    return execFileSync("git", ["diff", "--name-only", "HEAD~1", "HEAD"], { cwd: repo, encoding: "utf8" })
      .split("\n").map((l) => l.trim().replaceAll("\\", "/")).filter(Boolean);
  } catch (exc) {                              // fail LOUDLY — a silent [] here is graph drift
    throw new Error(`git diff failed: ${exc.message}`);
  }
}

Chunk 2 — regenerate the affected concept's Dependencies from the structural graph. WHAT: rebuild graph.json (the Track 2 extractor), then collect the touched service's outbound cross-service edges. WHY: this is pass 1 done deterministically — the facts come from the AST, so this section cannot hallucinate. GOTCHA: preserve the concept's frontmatter and prose; replace only the generated section, or the hook will eat human-written content.

def service_dependencies(graph: dict, module_prefix: str) -> list[str]:
    deps = set()
    for edge in graph["edges"]:
        src = edge["source"].split(":", 1)[1]
        dst = edge["target"].split(":", 1)[1]
        if src.startswith(module_prefix) and not dst.startswith(module_prefix):
            deps.add(f"- calls `{dst}` ({edge['provenance']})")   # provenance travels with the fact
    return sorted(deps)

def refresh_concept(bundle: Path, concept_rel: str, deps: list[str]) -> None:
    path = bundle / concept_rel
    path.parent.mkdir(parents=True, exist_ok=True)
    if path.exists():                          # surgical: replace ONLY the generated section
        head, sep, _tail = path.read_text(encoding="utf-8").partition("## Dependencies")
        base = head if sep else path.read_text(encoding="utf-8")
    else:
        name = Path(concept_rel).stem
        base = f"---\ntype: Service\ntitle: {name}\n---\n\n# {name}\n\n"
    block = "## Dependencies\n" + ("\n".join(deps) or "- (none detected)") + "\n"
    path.write_text(base.rstrip() + "\n\n" + block, encoding="utf-8")

function serviceDependencies(graph, modulePrefix) {
  const deps = new Set();
  for (const edge of graph.edges) {
    const src = edge.source.split(":")[1], dst = edge.target.split(":")[1];
    if (src.startsWith(modulePrefix) && !dst.startsWith(modulePrefix))
      deps.add(`- calls \`${dst}\` (${edge.provenance})`);  // provenance travels with the fact
  }
  return [...deps].sort();
}

function refreshConcept(bundle, conceptRel, deps) {
  const file = path.join(bundle, conceptRel);
  fs.mkdirSync(path.dirname(file), { recursive: true });
  let base;
  if (fs.existsSync(file)) {                   // surgical: replace ONLY the generated section
    const text = fs.readFileSync(file, "utf8");
    const idx = text.indexOf("## Dependencies");
    base = idx >= 0 ? text.slice(0, idx) : text;
  } else {
    const name = path.basename(conceptRel, ".md");
    base = `---\ntype: Service\ntitle: ${name}\n---\n\n# ${name}\n\n`;
  }
  const block = "## Dependencies\n" + (deps.join("\n") || "- (none detected)") + "\n";
  fs.writeFileSync(file, base.trimEnd() + "\n\n" + block, "utf8");
}

Chunk 3 — log, lint, and refuse to publish on failure. WHAT: append the dated log.md entry, run the M06/M07 lint, and exit non-zero if it fails. WHY: the log keeps knowledge history true mechanically; the lint gate is what makes the published bundle trustworthy. GOTCHA: the hook itself must not swallow this exit code — no nohup, no backgrounding, no || true. Every one of those turns a failed refresh into silent drift (M11 documents each, from real issue trackers).

#!/bin/sh
# orderflow enrichment hook — diff-scoped, loud on failure.
# No nohup, no backgrounding, no silent skips (see M11 for the postmortems).
set -e
REPO_ROOT="$(git rev-parse --show-toplevel)"
python3 "$REPO_ROOT/tools/refresh_concepts.py" "$REPO_ROOT" || {
    echo "[enrich] FAILED — bundle NOT updated. Fix before trusting the graph." >&2
    exit 1
}

Hands-On Exercise: Install and Break the Hook

📂 Get the files: labs/M08-enrichment-hook on GitHub — or clone the whole course once: git clone https://github.com/varasrinivas/knowledge-graph-course.git

What you'll build: the full hook from labs/M08-enrichment-hook/, installed into a git-initialized copy of orderflow — then broken on purpose, twice. Time: 45–60 min. Prerequisites: M06 and M07 labs.

1. Step 1 — Make sample-project a git repo.
   terminalCopy

   cd labs/sample-project
   git init -q && git add -A && git commit -qm "baseline"

2. Step 2 — Study, then install the hook. Read solution/refresh_concepts.py — the SERVICE_MAP table is the part you'd adapt to any real repo. Then copy solution/post-commit into .git/hooks/ (chmod +x on macOS/Linux).
3. Step 3 — Trigger it. Add a comment to services/billing/webhooks.py, commit, and watch the enrichment log lines appear.
   ✅ Checkpoint

   You see: changed-file detection → Dependencies refreshed with an edge count → log.md updated → lint result. If the hook didn't fire: exact filename post-commit, executable bit, correct hooks directory.

4. Step 4 — Break it (lint gate). Delete type: from a concept, commit any code change. The hook must FAIL loudly, naming the file, exit non-zero.
   ✅ Checkpoint

   If it passed, your lint gate is decorative — exactly the failure M11 is about. Fix the concept and re-commit.

5. Step 5 — Reflect. In one paragraph: which pipeline stages in your hook are deterministic, which would involve an LLM in a production pass 2, and why that split is what makes the bundle trustworthy.

Knowledge Check

Module Summary

What we built: orderflow's self-updating machinery — a post-commit hook that diff-scopes, refreshes Dependencies from the structural graph with provenance intact, logs, lints, and fails loudly.

Next module preview: the bundle exists and updates itself. Now it has to reach an agent efficiently. M09 moves to Track 4: serving graphs over MCP — one tool call instead of N file reads, and the honest arithmetic of what that saves.

References

• Google Cloud: How the Open Knowledge Format can improve data sharing (McVeety & Hormati, 2026)
• OKF Specification v0.1 — github.com/GoogleCloudPlatform/knowledge-catalog
• okf CLI — github.com/superops-team/okf · Kiso — github.com/oak-invest/kiso
• Course corpus: "Standardizing Agent Memory" (the pipeline blueprint); "Open Knowledge Format: A Complete Guide" (end-to-end walkthrough, pitfalls); "Production-Grade OKF + Graphify Setup" (what M11 inherits)