The 7 concepts of capstone work

Tap any concept to jump to its 5-card cluster.

• 1How to Approach a CapstoneThe five-phase methodology that prevents 73% of failures
• 2Project Selection GuidePick the right tier for your skill level
• 3The Three Domain AnchorsHealthcare, B2B Ecommerce, Public Records / UCC
• 4Mapping Skills to CapstonesWhich modules each project actually requires
• 5Self-Assessment RubricFive dimensions, scored 1–5, total out of 25
• 6Project Starter TemplateThe opinionated folder layout every capstone uses
• 7Evaluation HarnessThe mini test suite for AI agents

Five phases, in order, no skipping

A capstone is not "open editor, start coding." It is a five-phase pipeline: Requirements → Architecture → Build → Evaluate → Harden. Each phase has a checklist, and you do not move forward until the current phase is solid. Skip a phase and you pay 3× the time in rework.

The methodology exists because of one statistic: a 2024 survey found 73% of production agent failures traced back to skipped requirements analysis or architecture design. The agent worked on the developer's two test cases and exploded on the third. Phases 1–2 are not paperwork — they are the cheapest debugging you will ever do.

Building a house, not a Lego set

BEFORE: Picture two builders. Builder A grabs lumber and starts hammering. Builder B drafts blueprints, lays a foundation, frames the walls, runs plumbing, then inspects every joint before painting.

PAIN: Builder A's house works for the first family that walks in. The second family flushes the toilet and a load-bearing wall starts leaking. Now A is jackhammering finished tile to fix something that should have been planned in week one.

MAPPING: Capstone Phase 1 (Requirements) is the architect's brief. Phase 2 (Architecture) is the blueprint. Phase 3 (Build) is framing. Phase 4 (Evaluate) is the inspector. Phase 5 (Harden) is weatherproofing. Skip the brief and you ship a leaky house.

What each phase actually produces

1. Phase 1 — RequirementsRead the domain brief. Write the core problem in one sentence. Define measurable success ("95% correct on valid POs"). List 3–5 edge cases. Output: a one-page JSON spec. Prevents 80% of future bugs.
2. Phase 2 — ArchitecturePick the agent pattern (single-turn? ReAct? multi-agent?). Pick tools, data sources, state. Output: a one-page diagram + tool list. The cert exam rewards this exact reasoning.
3. Phase 3 — Iterative BuildBuild in vertical slices: one complete user flow end-to-end (happy path) before adding the next. NOT all tools first, then all prompts — that ships nothing for two weeks.
4. Phase 4 — EvaluationRun the test cases you wrote in Phase 1 through the harness. Score functionality. Find gaps. Fix gaps. Re-run. The harness, not your gut, says "done."
5. Phase 5 — Production HardeningAdd guardrails (M16–17), tracing (M19–20), cost optimization (M22). The difference between "works on my laptop" and "safe for users."

Which phase are you actually in?

# Honest self-check before opening the editor

IF you can't write success criteria in one sentence:
  YOU ARE IN Phase 1 # go back, write the spec

ELIF you don't know which agent pattern fits:
  YOU ARE IN Phase 2 # draw the diagram first

ELIF tests are passing on happy path only:
  YOU ARE IN Phase 3 # add edge-case slices

ELIF harness is red but you "feel" it works:
  YOU ARE IN Phase 4 # trust the score, not the vibe

ELIF harness is green but no tracing/guardrails:
  YOU ARE IN Phase 5 # not done yet

ELSE:
  SHIP IT # self-assess, then start the next capstone

Most "stuck" learners are in an earlier phase than they think. Going backwards one phase is faster than pushing forward.

Misconceptions

Pick the right trail rating for your toolbox

Five capstones, three difficulty tiers. Capstone 1 is a single-tool conversational agent (2–3 hours, requires Tracks 1–3). Capstones 2–3 are RAG and multi-tool orchestration (4–5 hours each, Tracks 1–5). Capstones 4–5 are multi-agent systems with production deployment (6–8 hours each, all 7 tracks).

Your "trail rating" is not how smart you are — it is how many tracks you have completed. Pick too easy, learn nothing. Pick too hard, finish nothing. Both waste your time. The selection guide tells you which trail you are actually ready for.

The trail-difficulty rating

BEFORE: Hikers don't pick trails by looking at the prettiest photo. They open the guidebook: "Easy — 2 miles, flat, families welcome" or "Difficult — 12 miles, 4,000 ft elevation, experienced hikers only."

PAIN: Take a family with toddlers on the expert trail and somebody is getting carried out by sundown. Put an expert on the family loop and they are bored stiff. Both are mismatched, and both are unhappy.

MAPPING: Capstone 1 is the family trail — gentle, complete, builds confidence. Capstone 5 is the alpine traverse — multi-day, exposure, every skill tested. Your "boots" are the tracks you have finished. Match the boots to the trail.

The three tiers, explained

1. Tier 1 — Capstone 1 (★)Single-tool conversational agent. 2–3 hours. Requires Tracks 1–3 (M01–M11). Tests whether you can combine the basics (prompt + tool + structured output) into something a user would actually use.
2. Tier 2 — Capstones 2–3 (★★★)RAG-powered domain expert (C2) and multi-tool orchestrator (C3). 4–5 hours each. Requires Tracks 1–5 (M01–M18). Adds retrieval, orchestration, guardrails, and the eval harness.
3. Tier 3 — Capstones 4–5 (★★★★★)Multi-agent systems (C4) and full production deployment (C5). 6–8 hours each. Requires all 7 tracks (M01–M22). Real enterprise complexity: coordination, observability, cost budgets.
4. Self-assessment factorsAsync familiarity. REST API experience. Domain comfort. Available time. Honest answers here pick the right tier.

Pick your capstone in 30 seconds

# Step 1: Check what you've completed

IF Tracks 1–3 NOT done:
  GO BACK — finish M01–M11 first

ELIF Tracks 1–3 done AND Tracks 4–5 NOT done:
  PICK Capstone 1 # 2-3 hrs, single-tool agent

ELIF Tracks 1–5 done AND Tracks 6–7 NOT done:
  PICK Capstone 2 OR 3 # RAG or multi-tool
  # C2 if your domain is doc-heavy; C3 if API-heavy

ELIF all 7 tracks done:
  PICK Capstone 4 OR 5 # multi-agent / production

# Step 2: Sanity check
IF tier feels "boring":
  ship it in < 2 hrs with score ≥ 20/25, then go up

IF tier feels "impossible":
  drop one tier — you're missing prerequisites

One well-executed capstone teaches more than five rushed ones. Quality over quantity.

Misconceptions

Three real industries, same agent skills

Every capstone runs in one of three domain anchors: A — Healthcare Pre-Authorization (CPT, ICD, HIPAA), B — B2B Ecommerce Order Tracking (POs, carriers, SLAs), or C — Public Records / UCC (entity resolution, Medallion Architecture). Same core skills (tool use, RAG, guardrails, eval) — different vocabulary, different stakes.

The domain is what turns a generic tutorial into a portfolio piece. "I built a recipe chatbot" is forgettable. "I built a clinical pre-auth agent that scores 87% on payer criteria with HIPAA-grade guardrails" is interview-worthy.

Three kitchens, one chef

BEFORE: Imagine three kitchens. A French restaurant with delicate sauces and precise temperatures. A sushi bar with raw fish and exacting knife skills. A food truck with speed-first cooking and a tiny counter.

PAIN: A cook who only knows ONE kitchen panics when handed a fish knife. The best chefs adapt their core skills — heat control, seasoning, timing — to any kitchen. That adaptability is what separates a cook from a chef.

MAPPING: Healthcare uses clinical terminology and strict compliance. Ecommerce uses real-time APIs and multi-party coordination. Public Records uses messy data and entity resolution. Different kitchens, same agent skills. The domain forces you to adapt.

What each domain actually demands

1. Domain A — Healthcare Pre-AuthDoctor's office asks insurer "will you cover this procedure?" Agent maps CPT codes (procedure) + ICD-10 codes (diagnosis) against clinical guidelines. Demands: structured output, RAG over policies, HIPAA-grade guardrails, human-in-the-loop for clinical review.
2. Domain B — B2B Order TrackingCustomer asks "Where's PO-2024-8847?" Agent queries ERP, warehouse, carrier API in one request. Demands: multi-tool orchestration, real-time aggregation, SLA alerting, multi-agent handoffs (sales → fulfillment → support).
3. Domain C — UCC / Public RecordsCredit analyst asks "what liens exist on Acme Corp?" Agent normalizes 50-state inconsistent filing formats and resolves entities. Demands: Medallion data pipeline (Bronze → Silver → Gold), RAG over legal docs, entity resolution as multi-step reasoning.
4. Same skills, different vocabularyTool use, RAG, guardrails, eval show up everywhere. The domain just decides which one you lean on hardest.

Pick your domain

# Match the domain to your real-world context

IF you work in healthcare, insurance, or pharma:
  PICK Domain A # compliance practice = career value

ELIF you work in retail, supply chain, or B2B sales:
  PICK Domain B # multi-tool orchestration is your daily reality

ELIF you work in fintech, legal, or data engineering:
  PICK Domain C # messy data + entity resolution = your bread and butter

ELIF none of the above (you're learning generally):
  PICK the UNFAMILIAR one
  # forces you to read briefs and ask clarifying
  # questions — the most transferable skill

# Once picked, STAY. Switching mid-project
# forfeits all the domain context you built up.

The mock data IS your dev environment. Pass evals on mocks first; only then swap in real APIs.

Misconceptions

The right tool for the right capstone

You've collected 22 modules of tools. The skills mapping tells you, for each capstone, exactly which modules to lean on. Capstone 1 uses 4 modules. Capstone 5 uses all 22. No project uses everything; every module is used by at least one project.

This matters because real-world agent work is selection, not memorization. Nobody hands you a list of which tools to use — you open the toolbox and decide. The skills mapping is your project-specific tool list, turning "I have 22 modules of knowledge" into "I need these 8 skills for this project."

The labeled toolbox

BEFORE: A carpenter's toolbox has dozens of tools, each in its own compartment — hammer, drill, level, measuring tape, safety goggles. Everything is organized, but nothing tells you which tool to use first.

PAIN: Walk up to a real project and you have to decide: hammer or drill? Measuring tape now or later? Choose wrong and you waste an hour using a hammer to drive a screw. Choose right and the job flows.

MAPPING: The skills mapping is the project-specific picking list taped inside the toolbox lid: "For Capstone 2, you definitely need the hammer (RAG) and drill (tool use). The measuring tape (eval harness) is recommended. The rest is optional."

What each capstone needs

1. Always required (every capstone)M01–M05: LLM mental model, tokens, prompts, structured output, function calling. The fundamentals are non-negotiable — everything builds on them.
2. Capstone 1 addsM08 (conversation management). Three skills total beyond the fundamentals. Deliberately small — tests whether you can compose basics into a working agent.
3. Capstones 2–3 addM06 (multi-tool orchestration), M09–M10 (RAG), M16–M18 (guardrails + eval). C2 = retrieval focus; C3 = orchestration focus. Same toolbox, different dominant tool.
4. Capstones 4–5 addM12–M14 (ReAct, planning, multi-agent), M19–M22 (tracing, monitoring, deployment, cost). The full stack. C5 deploys to actual cloud infra.
5. Domain shifts the emphasisHealthcare leans harder on M16–M17 guardrails. UCC leans harder on M09–M10 RAG. Same module list, different time allocation.

Build your readiness checklist

# Before Phase 1, score each module's readiness

required = modules_for(capstone, domain)
FOR EACH m IN required:
  status = SELF_RATE(m, scale=[done, in_progress, not_started])

IF any required.status == "not_started":
  GO BACK — review those modules first
  # attempting capstone with gaps = thrashing

ELIF any required.status == "in_progress":
  FINISH them OR pick a lower-tier capstone
  # half-learned = worse than not-learned

ELSE:
  YOU'RE READY — start Phase 1
  recommended_modules = PICK_TWO("strengthens weakest dimension")
  # add them as bonus learning during the build

Honesty matters. "I read M09" is not "I built a RAG pipeline." Self-rate by what you've shipped, not what you've watched.

Misconceptions

Five dimensions, scored 1–5, total of 25

The self-assessment rubric scores your capstone across five dimensions: Functionality, Code Quality, Prompt Engineering, Safety & Guardrails, Observability. Each scored 1 (needs work) to 5 (production-ready). Total ≥ 15/25 = competency. ≥ 20/25 = excellence.

Without a rubric, "is this good?" becomes pure vibes. With it, you get five concrete thermometer readings — honest data on strengths and gaps, so you know exactly what to improve before the next capstone.

The cooking thermometer

BEFORE: A recipe says "cook chicken until done." Without a thermometer, "done" is a guess. You cut into it, peek, recook, cut again — and end up with overcooked rubber or undercooked salmonella.

PAIN: Same thing happens to capstones without a rubric. You finish the agent and ask "is this good?" Without measurable criteria, you either think it's perfect (handled your one test case) or terrible (comparing to teams of 10).

MAPPING: The rubric is five thermometers — one per dimension. Each gives you a temperature reading from 1 to 5. You don't need all 5s. You need honest readings, so you know which dimension is undercooked and what to fix next.

What each dimension actually scores

1. Functionality1 = doesn't run. 3 = handles happy path. 5 = handles edge cases, malformed inputs, unexpected tool responses gracefully. The harness gives you this number.
2. Code Quality1 = spaghetti. 3 = reasonable modules and naming. 5 = production-ready with tests, types, clean separation of concerns.
3. Prompt Engineering1 = vague one-liner. 3 = structured prompts with examples. 5 = adversarial-tested prompts with structured-output schemas and fallback strategies.
4. Safety & Guardrails1 = none. 3 = input validation + error messages. 5 = PII redaction, rate limiting, circuit breakers, audit logging.
5. Observability1 = no logging. 3 = structured logs with request IDs. 5 = full tracing (Langfuse), cost tracking, monitoring dashboard.

Total ≥ 15/25 = competent. ≥ 20/25 = excellent. Be honest — this is for you, not external grading.

Score, then decide what to fix next

# After each capstone: score, then prioritize

scores = {
  func: RATE(1..5),
  quality: RATE(1..5),
  prompts: RATE(1..5),
  safety: RATE(1..5),
  observ: RATE(1..5),
}
total = SUM(scores)

IF total < 15:
  DON'T move to next capstone yet
  fix the lowest-scoring dimension first

ELIF total >= 15 AND total < 20:
  competent — ship + advance
  weakest_dim = MIN(scores)
  review modules for weakest_dim before next capstone

ELIF total >= 20:
  excellent — ship + advance one tier
  # you're ready for harder constraints

# Never average across dimensions to hide a 1/5
# A 5/5/5/5/1 is not "competent" — it's dangerous

A 1/5 in any dimension is a stop-the-line condition, even if your average looks fine. Fix the lowest before celebrating the highest.

Misconceptions

Same folder layout, every capstone

Every capstone starts from the same opinionated layout: src/ for agent code, data/ for domain mocks, eval/ for test cases and the harness, traces/ for observability output. Capstone 1 might have 3 files in src/; Capstone 5 might have 12 — the structure stays identical so you always know where things live.

The template is a starting point, not a cage. It imposes ZERO runtime constraints (unlike a framework). It is just a pre-organized empty notebook with labeled tabs — rename folders, add files, restructure freely. Saves an hour of boilerplate per capstone.

The pre-labeled notebook

BEFORE: Imagine starting every class with a brand-new blank notebook. You waste the first 10 minutes labeling sections: "Notes," "Homework," "Vocabulary," "Tests." Every class. Every notebook.

PAIN: Multiply that by five capstones and you've spent an hour creating folder structures instead of building agents. Worse, every capstone's structure differs slightly — so when you come back to C1 from C3, you can't find anything.

MAPPING: The starter template is a notebook where the section dividers are already in place. You open it and write. C1 looks like C5 looks like C3 — the muscle memory of "where does the loader go?" carries forward.

The four parts

1. Directory structureFive top-level folders: src/ (agent + tools + prompts + guardrails + config), data/ (per-domain mocks), eval/ (test_cases.json + harness + rubric), traces/ (observability output), plus README + .env.example at the root.
2. ConfigurationAn .env.example for API keys + model IDs, plus src/config.py for constants. Never commit real keys; the example file shows what's expected.
3. Domain data loaderPer-domain Python/TS module that simulates realistic API responses from in-memory dicts. Same function signature whether it's healthcare, ecom, or UCC — only the data shape differs.
4. Evaluation harnessThe mini test suite (covered next cluster). Pre-wired to run from eval/, score against the rubric, output a JSON report.
5. Structured error contractEvery tool returns {is_error, error_category, is_retryable, context}. The single most important design decision — the difference between "something broke" and "PO not found, retry won't help."

Pseudocode

The folder layout and the structured-error contract every tool returns:

# Folder layout (identical across capstones)
capstone-project/
  README.md            # project + setup + rubric
  .env.example         # API keys template
  src/
    agent.{py,ts}      # main loop
    tools.{py,ts}      # your functions
    prompts.{py,ts}    # system prompts
    guardrails.{py,ts} # input/output validation
    config.{py,ts}
  data/
    domain_a/          # healthcare mocks
    domain_b/          # ecom mocks
    domain_c/          # UCC mocks
  eval/
    test_cases.json
    harness.{py,ts}
    rubric.{py,ts}
  traces/              # logs + traces output

# Structured-error contract (every tool, every domain)
FUNCTION any_tool(args):
  TRY:
    result = do_lookup(args)
    IF NOT found:
      RETURN {is_error: true,
              error_category: "not_found",
              is_retryable: false,
              context: "PO X does not exist"}
    RETURN {is_error: false, ...result}
  CATCH e:
    RETURN {is_error: true,
            error_category: "internal_error",
            is_retryable: true, context: str(e)}

The error contract is what lets the agent say "that order doesn't exist, retrying won't help" instead of "something broke." Adopt it before writing tool #2.

Misconceptions

A mini test suite for AI agents

The evaluation harness runs your agent against pre-defined test cases and scores results automatically. Each test case specifies an input message, which tool the agent should (or should not) call, and what the response should contain. The harness compares actual to expected and produces a pass-rate number you can plot over time.

Without it, you test with 2–3 manual queries, declare victory, and discover crashes during self-assessment. With it, you make a code change, rerun the harness in seconds, and immediately see if you broke something. It also gives you the concrete pass-rate that becomes your Functionality score.

The car safety inspection

BEFORE: You can drive a car around the block once and call it "working." Headlights came on, brakes engaged, no warning lights. Looks fine.

PAIN: Then you take it on a highway in rain at night and discover the wipers don't switch to high speed, the rear brake light is out, and the headlights aim too low. The "around the block" test missed every real failure mode.

MAPPING: Manual testing is "around the block." The harness is the safety inspection — pre-defined checks, run identically every time, covering scenarios you wouldn't think to try in the moment. It catches the regression you didn't notice you caused.

The three-step loop

1. Define test casesEach one has: an input string, the expected tool call (or null = should NOT call any tool), should-contain phrases, should-NOT-contain phrases (errors, tracebacks), and a rubric tag (functionality or safety).
2. Run the agent against each caseHarness feeds input to your agent, captures the response and the list of tools called. If the agent crashes, that's an automatic fail with the error logged.
3. Score and tagFor each case, check tool usage + content matches. Pass/fail. Aggregate into a pass-rate. Tag-aware totals roll up into your rubric scores (functionality cases → Functionality dimension; safety cases → Safety dimension).
4. Aim for 8–10 cases minimum3 happy-path, 3 edge-case, 2–3 safety/error. Three is enough to prove it CAN work; eight starts to prove it RELIABLY works.
5. Run on every changeSeconds to rerun. Catches regressions before self-assessment. The harness IS your CI for capstones.

Pseudocode

The eval loop — one test case at a time, scored against three checks:

# Test case shape
test_case = {
  id: "TC-001",
  input: "What is the status of PO-2024-8847?",
  expected: {
    should_call_tool: "get_order_status",
    should_contain: ["shipped", "FedEx"],
    should_not_contain: ["traceback", "undefined"],
  },
  rubric_dimension: "functionality",
}

# The eval loop
FUNCTION run_evaluation(agent, test_cases):
  results = []
  FOR EACH tc IN test_cases:
    TRY:
      response, tools_called = agent(tc.input)
      passed = (
        CHECK_TOOL(tools_called, tc.expected) AND
        CHECK_CONTAINS(response, tc.expected) AND
        CHECK_NOT_CONTAINS(response, tc.expected)
      )
      results.append({id: tc.id, passed, dimension: tc.rubric_dimension})
    CATCH e:
      results.append({id: tc.id, passed: false, error: e})

  RETURN {
    pass_rate: SUM(passed) / LEN(results),
    by_dimension: GROUP_BY(results, "dimension"),
  }

The dimension grouping is what feeds your rubric. Functionality cases → Functionality score. Safety cases → Safety score. Auto-rubric for the dimensions a harness can measure.

Misconceptions

One question per concept

Tap a card to reveal the answer.