M11 · Production Operations: Drift, Staleness & Observability

Learning Objectives

Graphs Don't Self-Heal

Open with the scenario this entire module defends against. Your coding agent confidently refactors a payments module using a knowledge graph that is three weeks out of date — and the system never warned anyone. Every update hook reported success. Every dashboard was green. The agent's mental model quietly diverged from reality, and you found out when the refactor shipped.

The root cause is a single sentence you first met in M09's cache trap, now generalized: Graphify builds a static index. It does not update automatically as developers commit code. Without hooks explicitly wired in — and working correctly — the graph silently drifts from the codebase, and nothing in the tool's happy-path output tells you it's happening.

This is not hypothetical. In one reported case, GRAPH_SUMMARY.md — the ~700-token "cheap orientation" file handed to agents before a task — was measured two weeks stale on origin/main: last updated May 9, checked May 23. For that entire window it was served to agents as ground truth. As one GitHub issue discussion put it: "An agent reasoning over a stale bundle gives confidently wrong answers. If you refactor a module and forget to rerun /graphify, your agent is navigating with a stale map."

Anatomy of a Silent Failure: Four Documented Mechanisms

Everything in this section comes from real issue-tracker reports on the tools you've been using — it reads like a postmortem because it effectively is one. Four independent mechanisms, each capable of stopping rebuilds while producing no visible error.

1 — Windows hooks fail silently: the missing nohup

The post-commit and post-checkout hooks that trigger rebuilds use nohup to detach the rebuild process. Git for Windows' shell doesn't have nohup. The rebuild simply doesn't happen — and no error surfaces to the developer, because the hook framework doesn't treat a missing binary as a failure worth reporting loudly. Every Windows developer on the team commits daily, believes the graph updates, and is wrong. (This is why the M08 lab's hook deliberately avoids nohup and backgrounding entirely: loud and synchronous beats quiet and detached.)

2 — The extension allowlist drifted from the source of truth

The hook decides whether a commit warrants a rebuild by checking changed files against a hardcoded CODE_EXTS extension list. That list drifted out of sync with the authoritative list in graphify/detect.py. Result: commits touching valid, parseable code files silently skip triggering a rebuild, because the hook's gate logic doesn't recognize the extension. Note the shape of this bug — two copies of one fact, maintained separately, diverging. You'll see it again in the artifact-desync section, and it's the same disease OKF's "path is the identity" rule (M06) was designed to prevent.

3 — Resource gating turns "busy" into "stale, indefinitely"

Rebuilds are gated on a resource check before firing:

def _resources_ok():
    cpu_ok = psutil.cpu_percent(interval=0.1) <= 50 * cpu_count_fraction
    mem_ok = psutil.virtual_memory().available >= 2 * 1024**3   # 2GB
    return cpu_ok and mem_ok

if not _resources_ok():
    # rebuild is silently skipped; next commit will retry
    return

Each line is reasonable in isolation — you don't want a graph rebuild fighting your CI job for CPU. But the skip is silent, and the recovery strategy is "the next commit will retrigger it." On a quiet branch, or a repo where commits are infrequent, that next commit might be days away. The graph doesn't just lag; it lags for an unbounded, unmonitored duration. A gate without a catch-up path converts a temporary condition (busy machine) into a permanent one (stale graph).

4 — Caching outlives regeneration

M09's trap, now in its production context: MCP server integrations cache graph.json at startup with no hot-reload. Even a perfectly successful graphify update may never reach the agent actually querying the graph until the server process restarts. Fresh graph on disk, stale graph in the agent's working memory, simultaneously — and no artifact anywhere records that the two have diverged.

Artifact Desync: When the Three Outputs Disagree

Recall from M04 that one Graphify run produces three artifacts: graph.json (machine-readable), graph.html (visual), and GRAPH_REPORT.md (narrative). Three outputs from one run — which means three things that can stop agreeing.

If graphify update refuses to overwrite an existing graph.json under certain conditions, the three can desync: the human-readable report telling one story, the machine-readable graph another, with no built-in cross-check to flag the mismatch. A reviewer skims GRAPH_REPORT.md and sees the new module documented; the agent queries graph.json and doesn't know it exists. Both are "the graph," and they disagree.

There's an operational cousin to this problem: if graphify-out/ is tracked in git, every regeneration dirties the working tree, which blocks clean CI/publish workflows. Teams that don't catch it early end up with commit-hygiene problems layered on top of freshness problems. The fix is one line in .gitignore — but it must be a decision, not an accident, because ignoring the artifacts also means your CI must rebuild them rather than trusting committed copies.

The Scale Wall: When Regeneration Itself Becomes the Bottleneck

Assume every hook fires correctly, every time. Regeneration still isn't free — and at scale it isn't even bounded.

The worst offender is centrality. Graphify's suggest_questions() calls betweenness centralityA graph metric scoring how often a node sits on shortest paths between other nodes — the god-node detector from M01. Computing it exactly requires touching a large share of all vertex-edge combinations., which is O(V×E). On a monorepo measured at 450,000 nodes and 690,000 edges, that's roughly 310 billion operations — and it's a single-threaded graph algorithm, not an embarrassingly parallel one, so more cores barely help. In practice: a full rebuild on a 96-core Xeon was killed after 114 minutes without completing, while the same class of rebuild took roughly 10 minutes on an Apple M3 Pro on a different repo. Read those two data points together: cost here is non-linear in repo shape and nearly unpredictable from hardware specs alone.

"So use incremental updates" — yes, and calibrate your expectations. One reported case measured ~0.8 seconds for a small incremental diff; another measured 10+ seconds for incremental rebuilds against a full real monorepo. "Incremental" means "smaller than a full rebuild" — a much weaker guarantee than "fast." If your mental model says incremental ≈ instant, a 10-second post-commit hook will surprise you exactly where it hurts: in every developer's commit loop.

What Production-Grade Actually Requires

Five requirements. Each one exists because of a specific failure you've now seen. Treat graph freshness as an observable property, not an assumption — that's the headline over all five.

1. Log rebuild success/failure explicitly, with timestamps, somewhere you'll actually see it. Never trust hook exit codes alone — doors 1–3 all exit clean. A one-line append to a rebuild log turns "did it run?" from a belief into a query.
2. Cross-check the three artifacts against each other AND against the latest commit. graph.json, GRAPH_REPORT.md, and graph.html should agree on when they were generated, and that timestamp should be recent relative to git log -1. A one-line CI comparison catches the two-week-stale scenario before an agent does.
3. Restart or hot-reload MCP servers after regeneration. Don't assume a running process picks up a freshly written graph.json — verify it, or bounce it (door 4).
4. Gate rebuilds on resources only WITH an explicit retry/alert path. A silent no-op under load is fine as a first line of defense; it is not fine as the only line. Pair the gate with a scheduled catch-up job, plus an alert when staleness exceeds a threshold you've consciously chosen (door 3's fix).
5. Verify you're above the ~500-file payoff floor at all. Below it, the pipeline is ceremony, not engineering. Measure your own repo before adopting wholesale.

Code Walkthrough: Build the Freshness Gate

Checklist items 2 and 4 are one small script. Let's build it in three chunks — this is the same gate you'll run in the lab, and the one the capstone wires into CI.

Chunk 1 — compare the graph's clock against git's

What: read graph.json's mtime and HEAD's commit time; fresh means the graph was generated at or after the last commit. Why: this single comparison mechanically detects every one of the four silent failures — however the rebuild died, the symptom is identical: the artifact predates HEAD. Gotcha: compare against commit time, not wall-clock age. A repo untouched for a month with a month-old graph is perfectly fresh.

import subprocess, sys
from pathlib import Path

def sh(args, cwd):
    r = subprocess.run(args, cwd=cwd, capture_output=True, text=True)
    if r.returncode != 0:
        raise RuntimeError(f"{' '.join(args)}: {r.stderr.strip()}")
    return r.stdout.strip()

def check_fresh(repo: Path, graph: Path, max_age_commits: int = 0) -> int:
    if not graph.exists():
        print(f"STALE: {graph} does not exist — run the extractor")
        return 1
    graph_mtime = int(graph.stat().st_mtime)
    head_time   = int(sh(["git", "log", "-1", "--format=%ct"], cwd=repo))
    if graph_mtime >= head_time:
        print("FRESH: graph generated at/after HEAD commit")
        return 0
    lag = int(sh(["git", "rev-list", "--count",
                  f"--since={graph_mtime}", "HEAD"], cwd=repo))
    if lag > max_age_commits:
        print(f"STALE: graph predates HEAD by {lag} commit(s)")
        print("  fix: rerun the extractor, then RESTART any MCP server "
              "still holding the old graph in memory")
        return 1
    print(f"FRESH ENOUGH: lags by {lag}, within threshold {max_age_commits}")
    return 0

import { execFileSync } from "node:child_process";
import { statSync, existsSync } from "node:fs";

const sh = (args, cwd) =>
  execFileSync(args[0], args.slice(1), { cwd, encoding: "utf-8" }).trim();

function checkFresh(repo, graph, maxAgeCommits = 0) {
  if (!existsSync(graph)) {
    console.log(`STALE: ${graph} does not exist — run the extractor`);
    return 1;
  }
  const graphMtime = Math.floor(statSync(graph).mtimeMs / 1000);
  const headTime = parseInt(sh(["git", "log", "-1", "--format=%ct"], repo));
  if (graphMtime >= headTime) {
    console.log("FRESH: graph generated at/after HEAD commit");
    return 0;
  }
  const lag = parseInt(sh(["git", "rev-list", "--count",
                           `--since=${graphMtime}`, "HEAD"], repo));
  if (lag > maxAgeCommits) {
    console.log(`STALE: graph predates HEAD by ${lag} commit(s)`);
    console.log("  fix: rerun the extractor, then RESTART any MCP server " +
                "still holding the old graph in memory");
    return 1;
  }
  console.log(`FRESH ENOUGH: lags by ${lag}, within threshold ${maxAgeCommits}`);
  return 0;
}

Chunk 2 — cross-check the artifacts

What: re-extract into a throwaway file and compare node/edge counts against the stored graph. Why: a timestamp can be fresh while the content is wrong (a partial rebuild, a refused overwrite) — the count comparison catches desync that clocks can't. Gotcha: clean up the throwaway file in a finally; a leftover .freshcheck.json becomes tomorrow's confusing fourth artifact.

import json

def cross_check(repo: Path, graph: Path, extractor: Path) -> int:
    stored = json.loads(graph.read_text(encoding="utf-8"))
    fresh_out = graph.with_suffix(".freshcheck.json")
    try:
        sh([sys.executable, str(extractor), str(repo),
            "--out", str(fresh_out)], cwd=repo)
        fresh = json.loads(fresh_out.read_text(encoding="utf-8"))
    finally:
        fresh_out.unlink(missing_ok=True)      # never leave a 4th artifact
    if (len(stored["nodes"]), len(stored["edges"])) != \
       (len(fresh["nodes"]),  len(fresh["edges"])):
        print(f"DESYNC: stored {len(stored['nodes'])}n/{len(stored['edges'])}e "
              f"vs fresh {len(fresh['nodes'])}n/{len(fresh['edges'])}e")
        return 1
    print("CROSS-CHECK OK: stored counts match a fresh extraction")
    return 0

Chunk 3 — wire it into CI, before anything trusts the graph

What: run the gate as a pipeline step ahead of any agent-consuming job. Why: placement is the point — the gate is worthless after an agent has already acted on a stale graph. Gotcha: a failing gate must block (exit nonzero), not warn. Warning-only freshness checks are ignored by their second week.

steps:
  - name: Graph freshness gate
    run: python ci/freshness_gate.py . --max-age-commits 0 --cross-check
  - name: OKF bundle lint          # trust gates travel in pairs (next section)
    run: python ci/validate.py knowledge
  - name: Agent-consuming jobs
    run: ...                        # only reachable if both gates passed

Walk it, step by step

The code changes, the graph does not, and an agent answers confidently from a map that is now a lie. Watch the gate catch it — and watch the earlier step where nothing catches it at all.

OKF-Side Governance: Lint as the Trust Gate

The structural graph isn't the only artifact that rots. Recall OKF's deliberate leniency from M06: consumers must tolerate missing optional fields, unknown types, and broken links. That leniency is why adoption is easy — and it means a bundle can silently degrade into unlinked, out-of-date files without triggering any error anywhere. Drifted types ("API Endpoint" vs "Endpoint" vs "Route"), dangling cross-references, concepts nobody re-linked after a refactor: all of it is spec-conformant.

That's why lint holds the same position for the bundle that the freshness gate holds for the graph: run okf lint (13 built-in spec-compliance rules) as a hard CI gate before agents are allowed to act on the bundle. Given how lenient the spec itself is, lint is the only thing standing between "technically conformant" and "actually useful." The M08 pipeline already ends every enrichment pass with lint; CI re-running it is the backstop for edits that bypassed the pipeline.

The Defining Question

Everything in this module compresses into one test. Before you adopt Graphify+OKF — or before you audit a setup you already have — write down, concretely, how you would detect a stale graph within 24 hours.

Acceptable answers are mechanical: "CI runs the freshness gate on every push and blocks on lag > 0"; "a nightly job compares artifact timestamps to HEAD and pages past a 6-hour threshold"; "the MCP server exposes a status tool and the orchestrator refuses graphs older than the last commit." Unacceptable answers are beliefs: "the hooks handle it," "we'd notice," "it hasn't happened yet."

If you can't answer in specific, mechanical terms, you don't have a production-grade setup. You have a demo that hasn't failed yet. The real production question was never "is it 71× or 7×" — that's a number you quote in a meeting. The question that determines whether your setup survives contact with a real codebase is whether you'll know the graph went stale within a day, or find out in a month, the hard way, after an agent has shipped a confidently wrong refactor.

Hands-On Exercise

📂 Get the files: labs/M11-freshness-gate on GitHub — or clone the whole course once: git clone https://github.com/varasrinivas/knowledge-graph-course.git

What you'll build: the freshness gate, run green, then deliberately broken and caught. Time: 30–45 min. Lab folder: labs/M11-freshness-gate/. Prerequisite: the M08 lab (a git repo whose graph updates on commit).

cd labs/sample-project
python ../M11-freshness-gate/solution/freshness_gate.py . --max-age-commits 0

git commit --allow-empty -qm "commit without rebuilding the graph"
python ../M11-freshness-gate/solution/freshness_gate.py . --max-age-commits 0

- run: python labs/M11-freshness-gate/solution/freshness_gate.py . --max-age-commits 0 --cross-check

Stretch goals: add a staleness-age alert (page past N hours, not just N commits); add an MCP status tool to your M09 server and make the gate query it, closing the fresh-disk/stale-memory gap.

Knowledge Check

Module Summary

What we built on orderflow: a CI freshness gate that passes green, catches a deliberate no-rebuild commit with exit 1 and an actionable fix, and cross-checks artifact content — the mechanical answer to the 24-hour question.

Next module preview: the machinery is production-grade; the last lever is human. M12 covers structuring projects so agents don't get lost in the first place — layered CLAUDE.md files, skills, subagents, deny rules — and where this 90-day-old ecosystem goes next.

References

• "Production-Grade OKF + Graphify Setup, Not Just a Demo" — the drift postmortem and checklist (Data Science Collective, Aug 2026)
• Graphify issue tracker reports (silent hook failures, scale measurements)
• okf CLI — okf lint, 13 spec-compliance rules (github.com/superops-team/okf)
• Course labs: labs/M11-freshness-gate/, labs/M08-enrichment-hook/
• M01 (centrality), M04 (payoff floor), M09 (cache trap) — the concepts this module operationalizes