Large Java Codebase — Measure the Token Reduction

Learning Objectives

By the end of this capstone you will be able to:

• Extract a structural graph from a real Java codebase (624 to 9,000+ files) with tree-sitter-java — and read its provenance report critically.
• Verify a graph against ground truth before trusting any number it produces — the step no benchmark reader ever does.
• Serve a large graph to any MCP-aware agent with the parametrized M09 server (--graph).
• Run a two-arm token benchmark and produce your own measured table — per-question multiples, reduction percentages, exploration burden.
• Explain your numbers the way this course taught you: per-question vs session-level, ceiling case vs median, and why a smaller repo can show a bigger multiple.

Skill level: Intermediate → Advanced

The Brief: Flip the Floor

In Capstone 1 you built the full three-layer stack — structural graph, OKF bundle, enrichment hook, MCP server, freshness gate — over orderflow, a 15-file repo. Your measured improvement was a single-digit multiple, and the course told you that was the correct, expected result: orderflow sits far below the ~500-file threshold where graph tooling pays for itself.

That lesson only lands completely when you see the other side. This capstone flips the setup: you will graph Apache commons-lang (624 Java files) and Spring Framework (9,247 Java files — over 13 million tokens of source), then run a deterministic benchmark harness that measures, for real structural questions, what an exploring agent would pay versus what one graph query costs.

By the end you will hold a benchmark table with your name on it — the same shape of table that M04 and M05 taught you to read with suspicion. Producing one yourself, and then explaining its numbers honestly, is the graduation exercise of this course.

What you'll wire together

kg_extract_java.py  ──►  graph.json  ──►  MCP server (M09, --graph flag)
        │                                        │
   [Java repo]                          measure_tokens.py
   commons-lang     (624 files)                  │
   spring-framework (9,247 files)         your benchmark table
   synthetic        (offline fallback)    (per-question + aggregate)

Everything reuses course machinery: the graph schema from M01, the extraction approach from M03 (same tree-sitter family, new grammar), the M09 server with its new --graph flag, and — in the writeup — M11's staleness discipline.

Concept: The Two Arms of an Honest Measurement

Every benchmark comparing "agent with graph" to "agent without graph" is secretly a claim about what the without arm costs. Get that arm wrong and the multiple is fiction. So before you run anything, understand precisely what our harness charges each side.

Notice what the baseline is not: it is not what a lazy agent spends. A lazy agent reads three of the 46 matching files, answers from those, and is wrong about the other 43 call sites. The baseline is the mechanical floor of a correct answer — the honest denominator. That asymmetry (the graph is complete by construction; exploration is complete only if you pay for every file) is the entire economic argument of this course, made concrete.

Concept: When the Graph Should Refuse to Guess

Java breaks the bare-name resolution you used on Python in a specific way: overloading and interface convention mean the same method name is defined in many places. In orderflow, decode_jwt had one definition; in Spring, getName has hundreds.

The extractor keeps the three provenance tags you know from M01 — with one addition:

Why drop rather than emit? A call to a name defined 150 times carries no structural signal — emitting 150 candidate edges per call site is quadratic noise wearing an honesty costume. The first version of this extractor did exactly that and produced 183,568 edges for a synthetic repo whose true structure had under 6,000; the cap brought it back to 5,960 without losing a single seeded ground-truth edge. Refusing to guess is the honest answer here. Real tools do better not by guessing harder but by escalating to type resolution — M03's LSP ladderM03's resolution ladder: heuristic bare-name matching → LSP-backed disambiguation (ask the language server via textDocument/definition) → accept AMBIGUOUS. Each rung costs more and resolves more., where a language server like Eclipse JDT resolves the receiver's static type.

Concept: Reading Benchmarks Honestly — Now With Your Own

M04 gave you the skeptic's checklist for other people's benchmarks: vendor-reported vs replicated, ceiling vs median, corpus topology. Now apply it to the table you are about to produce. Three questions matter:

1. What level is the number? Per-question structural lookups (this harness, okf-rs's ~400x/query) vs whole sessions (CodeGraph's task benchmarks: OkHttp 645 files → 13%, Django ~3,000 → 36%, VS Code ~10,000 → 78%; independent Graphify replications 6.8x–49x). Both are true; they measure different things.
2. What does the baseline assume? Ours: complete confirmation of every call-site grep hit. Weaker baselines (agent samples a few files) produce smaller multiples and wrong answers; stronger claims usually hide a lazier baseline.
3. What was selected, and by whom? Symbols here are chosen by caller count, deterministically, before any cost is known. If you let the selector see grep expense first, you can make the aggregate say almost anything — try it in Going Further and watch the number inflate.

Bridge to the build: you now know what each arm charges, why some names get dropped, and how to classify the number that comes out. Time to produce it.

The Build

📂 Get the files: labs/CAPSTONE-2-java-token-benchmark on GitHub — or clone the whole course once: git clone https://github.com/varasrinivas/knowledge-graph-course.git

All paths below are relative to labs/CAPSTONE-2-java-token-benchmark/. The full packet, with the same steps, lives in that folder's README.md. Pick at least two repo sizes — the comparison across sizes is the lesson.

pip install tree-sitter tree-sitter-java

python ../shared_tools/kg_extract_java.py <repo> --out <repo>-graph.json --quiet

Rebuilt: 10648 nodes, 40309 edges -> commons-lang-graph.json
  nodes: {'class': 939, 'interface': 104, 'method': 8981, 'module': 624}
  edges: {'AMBIGUOUS': 32333, 'EXTRACTED': 2055, 'INFERRED': 5921}
  unresolved: 10258 calls to 36 over-ambiguous names (> 8 candidates) dropped,
              e.g. ['append', 'toString', 'get', 'of', 'contains']

Rebuilt: 110885 nodes, 432741 edges -> spring-graph.json
  nodes: {'class': 15806, 'interface': 1819, 'method': 84065, 'module': 9195}
  edges: {'AMBIGUOUS': 379024, 'EXTRACTED': 16022, 'INFERRED': 37695}
  unresolved: 228955 calls to 911 over-ambiguous names dropped, e.g. ['assertThat', 'get', 'getName']

This is the step no benchmark reader ever does — and the reason your numbers will deserve more trust than the ones on Medium.

import json
g = json.load(open("synth-graph.json")); truth = json.load(open("synth-java/known_truth.json"))
callers = [e for e in g["edges"] if e["target"].endswith(".lookup")]
print(len(callers), "found, expected", truth["lookup_callers"])  # 1192 found, expected 1192

# standalone
python ../M09-mcp-graph-server/solution/server.py --graph <path>

# register with your assistant (absolute paths!)
claude mcp add java-graph -- python /abs/path/server.py --graph /abs/path/graph.json

python solution/measure_tokens.py <repo> <graph.json> --questions 8 --json bench.json

who calls ...?           files baseline tok graph tok reduction multiple
--------------------------------------------------------------------------
getClass                  1340    4,075,945    84,885     97.9%    48.0x
registerBeanDefinition     174      764,534    46,540     93.9%    16.4x
isTrue                    1070    3,377,634   122,220     96.4%    27.6x
...
--------------------------------------------------------------------------
AGGREGATE                 4535   15,194,788   451,436     97.0%    33.7x

Measured Reference Results

All from real runs of the course tooling (2026-08-24 snapshots; extraction: commons-lang ~12s, Spring ~2m27s):

What a Real Live Run Looks Like

Everything above is the mechanical benchmark — no model in the loop. We also ran the Phase 6 live experiment: the same questions asked via claude -p against this spring-framework snapshot, Sonnet on every arm, medians of 3 independent fresh sessions per cell. Four arms: baseline (Read/Glob/Grep only, clean repo), init (identical tools, but every spring-* module first got a headless-generated CLAUDE.md — M12's layered-context pattern, i.e. the add lever, as a measured arm; one-time generation cost ~$3.40 for 22 modules), graph-available (graph MCP tools plus file tools, prompt says "prefer the graph"), and graph-forced (MCP tools ONLY — the Graphify-style forced integration).

Read the three arms as one story:

1. Forcing is what unlocks the win. With the graph as the only tool, the callers question was answered in ONE turn at 42K tokens — 64% under baseline. When the graph was merely available, the model queried it, distrusted the fan-out, and re-verified against files — paying for both paths. On forClass that made graph-available WORSE than having no graph at all (10 turns, 415K).
2. Fan-out is the graph's poison. Bare-name resolution merges overloads into 1,000+-caller answers; even forced, forClass only tied the baseline, with wild spread (45K–698K). A type-resolved graph with compact answers is what earns both trust and savings.
3. Logic questions invert the verdict. The graph-only agent DID answer both "why" questions correctly — it walked callee edges to reconstruct the override-guard chain and the post-processor re-scan — but at roughly DOUBLE the baseline's cost (L1: 524K vs 231K, 12 turns of edge-triangulation vs 6 turns with one file read). The graph can substitute for reading code, expensively; it cannot beat reading code at comprehension. Caveat: Spring is in the model's training data, so edge-walking "confirmed" logic it partly already knew — on a private codebase, graph-shaped confidence over unread code is a hallucination risk, not a capability.
4. The line-granular baseline is strong. A real agent's Grep returns matching lines, never whole files, so the mechanical whole-file floor (the 33.7x above) vastly overstates what live exploration costs.
5. The add lever is a per-session wash that hides texture. The init arm landed at +2% overall — but that average conceals a 30% WIN on one navigation question (the CLAUDE.md map sent the agent straight to the right module: 2 turns) and a 58% LOSS on another (module CLAUDE.mds load into context when the agent works in those directories, and on that question the added context never paid for itself). CLAUDE.mds are conventions-and-orientation context, not a lookup index — M12's "facts every session needs" rule, measured. The one-time $3.40 generation cost amortizes only if many future sessions hit the winning pattern.
6. The synthesis is M10's router. Neither arm wins everywhere: forced-graph is best for structural lookups (−26%), files are best for comprehension (forced-graph +120% there). A router sending each question class to its cheap arm would have spent ~523K on these four questions vs 585K all-baseline — the hybrid-stack lesson, measured.
7. Measurement anomaly, reported not hidden: the forced arm's dollar cost read high relative to its token counts (the CLI's cost field didn't reconcile with its usage fields on those runs); tokens and turns are the trustworthy columns there.

The Four Scenarios, In Depth

Each arm of the live benchmark is a deliberate integration scenario you might actually ship. This is the full reference for each: what it simulates, its exact configuration, how to run it, what it measured, and the insight it bought. (Same content ships as live_bench/SCENARIOS.md next to the tooling.)

Scenario 1 — baseline: a normal agent, nothing added

Simulates: the out-of-the-box experience — file tools, no curated context. The control every other scenario is measured against.

Insight: far stronger than the mechanical benchmark predicts, because a real agent’s Grep returns matching lines, not whole files — 3 turns on the callers question without wholesale file reads. Anything that wants to win has to beat line-granular grep, not the whole-file strawman.

Scenario 2 — init: curated CLAUDE.md context (the ADD lever)

Simulates: a team that invested in per-module context files — M12’s layered-CLAUDE.md pattern. Claude Code auto-loads a module’s CLAUDE.md when the agent works in that directory; the orientation rides along, and so does its token cost.

Insight: the +2% wash hides the story — a 30% WIN where the CLAUDE.md map sent the agent straight to the right module (81K, 2 turns), a 58% LOSS where the auto-loaded context never paid off. CLAUDE.mds are orientation-and-conventions context, not a lookup index; the generation cost amortizes only across sessions that hit the winning pattern.

Scenario 3 — available: graph offered, not imposed

Simulates: the most common real integration — a graph MCP server registered alongside normal tools, with guidance to prefer it. The agent chooses.

Insight: optionality actively hurts with a fan-out-heavy graph. The model queries the graph, receives a 1,000+-candidate bare-name answer, rationally distrusts it, and re-verifies against files — paying for BOTH retrieval paths (10 turns, 415K on forClass). “We registered a graph and told the model to prefer it” is not an integration strategy.

Scenario 4 — forced: graph as the only tool

Simulates: a Graphify-style forced integration — the graph is the single search surface, no file access.

Insight (two): forcing is what unlocks the structural win — remove the option to double-check and one-call answers finally beat grep. And the forced arm is a canary: our first two campaigns measured a “graph arm” whose server was silently crashing; with file tools as fallback the failure was invisible. The forced arm, with no fallback, reported “tools unavailable” and exposed it. Smoke-test a forced session before trusting any graph-arm number. On logic questions it answered correctly by edge-walking — at 2x the cost, with the training-data confound (on private code, graph-shaped confidence over unread code is a hallucination risk).

The small-repo counterpoint: the same four scenarios on 36 files

We reran the identical four-scenario campaign (same runner, same methodology, questions adapted to the domain) on priorauth-api-boot3 — a 36-file Spring Boot 3 prior-authorization API. The mechanical harness said 15.2x per-question. The live grid said something else entirely:

The verdict flipped. On spring-framework, forced-graph won structural and init was a wash. Here, one 55-line root CLAUDE.md describing the submission→determination flow is the best arm overall (−10%, and −38% on the MANUAL_REVIEW question — the map pointed straight at CriteriaScoringService), while the graph loses in every configuration — even forced (+84%): at 36 files grep costs 2 turns, the graph is tiny and half-AMBIGUOUS, and forced edge-walking on logic questions burned 17–18 turns.

This is the ~500-file payoff floor (M04, M11) measured live instead of asserted, and the sharpest form of the course’s closing lesson: which context lever wins is a property of the repo and the question mix, not of the tool. Big haystack + structural questions → forced graph. Small repo one person can hold in their head → one good CLAUDE.md beats everything. The only way to know which world you are in is the measurement you just learned to run. (Raw 48-session dataset: expected_output/live-run-priorauth.json.)

Walk it, step by step

The four arms walked one at a time on the same question set. Watch for the flip at step 6: the arm that wins structural questions by 26% loses logic questions by 120% — and then step 7, where on a 36-file repository every graph arm loses to a single hand-written CLAUDE.md.

Run These Benchmarks Yourself

Everything in the previous section is reproducible with the tooling that ships in labs/CAPSTONE-2-java-token-benchmark/live_bench/. The complete scenario reference — what each of the four arms simulates, its exact configuration, step-by-step commands, measured findings, and pitfalls — lives alongside the tooling in live_bench/SCENARIOS.md. The workflow below is the distilled version of what actually happened when we ran it — including the guardrails added after each thing that went wrong.

The tooling

The four arms

Documented in depth in The Four Scenarios, In Depth above — simulation intent, exact configuration, per-arm steps, measured results, and pitfalls. The short version: run baseline first on a clean repo, init only after generating CLAUDE.mds (and clean up after), available/forced need --mcp-config, and forced doubles as your server canary.

The run, step by step

cd labs/CAPSTONE-2-java-token-benchmark/live_bench

# 0. Prereqs: graph built (Phase 1) and the M09 server smoke-tested (Phase 3).
#    DO NOT SKIP THE SMOKE TEST — see the canary box below.

# 1. Control first, clean repo:
python run_live_bench.py D:/tmp/spring-framework --arms baseline --reps 3

# 2. The graph arms (fill in mcp-config from the template first):
python run_live_bench.py D:/tmp/spring-framework --arms available,forced --reps 3 --mcp-config mcp.json

# 3. The init arm — write the CLAUDE.mds, run, then clean up:
python init_claude_mds.py D:/tmp/spring-framework --prefix spring-
python run_live_bench.py D:/tmp/spring-framework --arms init --reps 3
python init_claude_mds.py D:/tmp/spring-framework --remove

# 4. The table:
python aggregate.py runs

Methodology rules (each one bought with a mistake)

1. Fresh sessions, medians of ≥3. Identical sessions varied up to 2x in our runs; a single run is a coin flip. Every claude -p call is a fresh session — repetition is the whole methodology.
2. Arm order matters. Baseline before init (the runner records the repo's CLAUDE.md count in every run and warns when a "baseline" run is contaminated).
3. The forced arm is your canary. Our first two campaigns measured a graph arm whose MCP server was silently crashing — invisible because file tools gave it something to fall back on. The forced arm has no fallback: if the server is broken, forced sessions say so instead of producing plausible garbage.
4. Same model everywhere, one variable per arm. The runner pins the model; the only thing that changes between arms is tools/context.
5. Windows gotcha: prompts must be single-line — the .cmd shim truncates argv at embedded newlines (the runner handles this; keep it in mind if you edit it).
6. Budget before you start: at Sonnet, expect roughly $0.10–0.40 per session, ~$2.50 per 3-rep arm campaign on a repo this size, plus ~$3 for a 22-module init pass.

Test Cases

Going Further OPTIONAL

• Wire the freshness gate: point your Capstone-1 freshness_gate.py at the Spring graph in CI; commit without re-extracting and watch it block.
• Benchmark the benchmark: modify the selector to rank by grep expense instead of caller count, re-run, and watch your aggregate inflate — you have now manufactured a ceiling case on purpose, which inoculates you against ever being fooled by one.
• Compare instruments: run Graphify or CodeGraph over commons-lang and diff their edge sets against kg_extract_java's — expect them to win on resolution (they do type analysis; you cap at 8 candidates).
• Extend the server: add extends/implements queries (graph_api-style) — the edges are already in graph.json.
• Stress the baseline: add a --noise flag to gen_java_repo.py (dead code, comments naming symbols) and measure how the exploration arm degrades while the graph arm doesn't.

Knowledge Check

Summary

What you built: a verified Java structural graph pipeline, an MCP serving path for any pre-built graph, and — the real artifact — a benchmark table you can defend line by line. That closes the course: you can now build the map, keep it honest, and measure exactly what it saves.

References

• Lab packet: labs/CAPSTONE-2-java-token-benchmark/README.md (targets, phases, expected outputs)
• Tools: labs/shared_tools/kg_extract_java.py · gen_java_repo.py · solution/measure_tokens.py · M09 server --graph flag
• tree-sitter Java grammar — github.com/tree-sitter/tree-sitter-java
• Apache commons-lang — github.com/apache/commons-lang · Spring Framework — github.com/spring-projects/spring-framework
• Course modules this builds on: M03 (parsing & the LSP ladder), M04/M05 (benchmark literacy), M09 (MCP serving), M11 (staleness)