M23: Capstone Project Series

Learning Objectives

• Apply a structured five-phase methodology (Requirements → Architecture → Build → Evaluate → Harden) to build an agent from scratch
• Select the appropriate capstone project tier based on your skill level and completed modules
• Understand how the same agent patterns (tool use, RAG, guardrails) adapt to three different industry domains
• Use a self-assessment rubric to evaluate your capstone across functionality, code quality, prompts, safety, and observability
• Set up and run a project starter template with domain-specific data loaders and evaluation harness

How to Approach a Capstone Project

Here is what a Phase 1 requirements document actually looks like for a capstone project — this is the kind of artifact you would create before writing any code:

Project Selection Guide

Here is what the "trail rating" actually looks like for each capstone tier:

The Three Domain Anchors

Here is what "same skill, different domain" looks like in practice — the same tool-use pattern adapted to three kitchens:

Domain A: Healthcare Pre-Authorization

Pre-authorizationA process where healthcare providers must get approval from insurance companies (payers) before delivering certain services like surgeries, imaging, or specialty drugs. Also called "prior auth" or "pre-auth." is the process where a doctor's office asks an insurance company "will you cover this procedure for this patient?" before performing it. The agent must evaluate clinical requests against insurance criteria.

Key data: The agent works with two main code systems. First, CPT codesCurrent Procedural Terminology codes — 5-digit numbers that identify specific medical procedures. For example, CPT 27447 = total knee replacement. Used for billing and pre-authorization. identify the procedure (e.g., CPT 27447 = total knee replacement). Second, ICD-10 codesInternational Classification of Diseases, 10th revision — alphanumeric codes that identify diagnoses. For example, M17.11 = primary osteoarthritis of the right knee. Used alongside CPT codes in pre-authorization. identify the diagnosis (e.g., M17.11 = osteoarthritis of the right knee). Beyond codes, the agent also consults payer policy documents, clinical guidelines, and formulary lists.

Why it is great for agents: This domain exercises nearly every agent skill. You will use multi-step decision logic to evaluate clinical criteria. You will use RAG to retrieve relevant policy documents. You will produce structured output for authorization forms. You will implement human-in-the-loop flows for clinical reviewer approval. And you will need strict guardrails for HIPAAHealth Insurance Portability and Accountability Act — US law that sets standards for protecting sensitive patient health information (PHI). Violations can result in fines from $100 to $50,000 per violation. compliance and PHI handling — all under real cost and time pressure.

Domain B: B2B Ecommerce Order Tracking

B2B ecommerce involves complex order lifecycles — from RFQRequest for Quote — a formal document sent by a buyer to suppliers asking for pricing and terms on specific products or services. The starting point of many B2B transactions. and POPurchase Order — a legally binding document issued by a buyer to a seller, specifying the items, quantities, and agreed prices. Unlike B2C "orders," POs involve approval workflows and contract terms. creation through multi-warehouse fulfillment, partial shipments, and carrier tracking. Unlike B2C, B2B orders have approval workflows, net payment terms, volume discounts, and multi-stakeholder communication.

Key data: The agent juggles several data sources. SKUs and PO numbers identify products and orders. Carrier tracking APIs (FedEx, UPS, DHL) provide shipment status. Warehouse management system data shows inventory and fulfillment progress. ERP order status tracks the overall lifecycle. And SLAService Level Agreement — a commitment between a provider and a customer about delivery timelines, quality standards, and penalties for violations. For example, "all orders ship within 48 hours or the customer receives a 5% discount." commitments define the deadlines your agent must flag when they are at risk.

Why it is great for agents: This domain forces you to orchestrate multiple tools at once — querying an ERP system, a warehouse management system, and a carrier API in a single request. You will build real-time status aggregation across these sources. You will implement proactive alerting for delayed shipments. And you will practice multi-agent handoffs (sales → fulfillment → support) with production-grade observability.

Domain C: Public Records / UCC Data Engineering

UCC filingsUniform Commercial Code filings — public records documenting secured commercial transactions (liens on business assets). Filed with US Secretaries of State. Used in credit risk profiling and due diligence. are public records that document who has a financial claim (lien) on a company's assets. Banks, investors, and credit analysts use them to assess risk before lending money or making deals.

Key data: The core records are UCC-1 financing statements (who has a lien on what) and UCC-3 amendments that modify, continue, or terminate those liens. Each filing includes debtor and secured party names and addresses, collateral descriptions, filing dates, and lapse dates. The challenge? This data comes from 50+ state offices in wildly inconsistent formats — CSV, fixed-width text, XML, and sometimes PDF.

Why it is great for agents: This domain tests three distinct skill groups. First, data engineering pipeline orchestration using the Medallion ArchitectureA data pipeline pattern with three layers: Bronze (raw ingestion), Silver (cleaned and normalized), Gold (business-ready aggregations). Named after medal tiers — each layer adds more value and structure to the data. (Bronze → Silver → Gold). Second, RAG on legal and regulatory reference documents to answer compliance questions. Third, entity resolutionThe process of determining whether two records refer to the same real-world entity. For example, "Acme Corp," "ACME CORPORATION," and "Acme Corp Inc" might all be the same company. This is a multi-step reasoning task, not a simple string match. as a multi-step reasoning task — determining whether "Acme Corp" and "ACME CORPORATION" are the same entity. You will also practice evaluation and testing focused on data quality metrics.

Mapping Course Skills to Capstone Projects

Here is what a skills mapping actually looks like in practice — a concrete checklist you would create before starting Capstone 2:

Self-Assessment Rubric

Here is what a completed self-assessment actually looks like — a JSON report you would generate after finishing your capstone:

Project Starter Template

Every capstone project starts from the same directory structure. This template gives you configuration files, domain data loaders, and placeholder modules so you can focus on building the agent — not the scaffolding. Without a template, you would spend the first hour of every capstone creating folders, writing boilerplate, and setting up imports. The template eliminates that friction.

The template is intentionally opinionated about organization. Your agent code lives in src/, your mock data lives in data/, and your evaluation tests live in eval/. This separation matters because it scales: Capstone 1 might have 3 files in src/, while Capstone 5 might have 12 — but the structure stays the same. You always know where to find things.

How does a starter template differ from a framework or boilerplate generator? A framework imposes runtime constraints — you must use its request router, its plugin system, its lifecycle hooks. The starter template imposes zero runtime constraints. It is just a recommended folder layout plus domain-specific mock data. You can rename folders, add files, or restructure however you want. The template is a starting point, not a cage. Think of it as a pre-organized empty notebook with labeled tabs, not a form you must fill in.

The starter template has four parts: the directory structure, the configuration, the domain data loader, and the evaluation harness. Let's walk through each one.

Directory Structure

This is the recommended layout. Every file has a clear purpose, and the structure scales from Capstone 1 (simple) to Capstone 5 (complex).

capstone-project/
├── README.md                  # Project description, setup, and rubric
├── requirements.txt           # Python dependencies
├── .env.example               # Template for environment variables
├── src/
│   ├── agent.py               # Main agent implementation
│   ├── tools.py               # Tool definitions (your functions)
│   ├── prompts.py             # System prompts and templates
│   ├── guardrails.py          # Input/output validation
│   └── config.py              # Configuration and constants
├── data/
│   ├── domain_a/              # Healthcare: CPT codes, policies
│   ├── domain_b/              # Ecommerce: PO database, SKUs
│   └── domain_c/              # UCC: filings, entity records
├── eval/
│   ├── test_cases.json        # Evaluation scenarios
│   ├── harness.py             # Automated evaluation runner
│   └── rubric.py              # Self-assessment scoring
└── traces/                    # Observability output (logs, traces)

capstone-project/
├── README.md                  # Project description, setup, and rubric
├── package.json               # Node.js dependencies
├── tsconfig.json              # TypeScript configuration
├── .env.example               # Template for environment variables
├── src/
│   ├── agent.ts               # Main agent implementation
│   ├── tools.ts               # Tool definitions (your functions)
│   ├── prompts.ts             # System prompts and templates
│   ├── guardrails.ts          # Input/output validation
│   └── config.ts              # Configuration and constants
├── data/
│   ├── domain_a/              # Healthcare: CPT codes, policies
│   ├── domain_b/              # Ecommerce: PO database, SKUs
│   └── domain_c/              # UCC: filings, entity records
├── eval/
│   ├── test_cases.json        # Evaluation scenarios
│   ├── harness.ts             # Automated evaluation runner
│   └── rubric.ts              # Self-assessment scoring
└── traces/                    # Observability output (logs, traces)

Domain Data Loader

Each domain has a data loader that provides mock data for development and testing. You do not need real healthcare APIs or carrier tracking services — the loaders simulate realistic responses so you can focus on building agent logic.

Let's build the Domain B (Ecommerce) data loader as an example. This loader simulates two things your real agent would talk to: an ERP database (where order details live) and a carrier tracking API (where shipment status lives). In production, these would be real HTTP calls to FedEx or your company's ERP system. For the capstone, we use in-memory dictionaries with realistic data instead. The same pattern applies to Domain A (healthcare policies and clinical criteria) and Domain C (UCC filings and entity records) — swap the data structure, keep the same function signature pattern.

Pay attention to the error response structure — this is the most important design decision in the loader. Every function returns a dictionary with three fields: is_error (did something go wrong?), error_category (what kind of problem?), and is_retryable (should the agent try again?). This is the structured error pattern from M05, and it is the difference between an agent that says "something broke" and one that says "that order doesn't exist, and retrying won't help — let me tell the customer." Without these fields, your agent is flying blind when tools fail.

# data/domain_b/loader.py
# Domain B: B2B Ecommerce Order Tracking — Mock Data Loader
# Simulates an ERP order database and carrier tracking API
# so you can develop your agent without real API credentials.

import json
from datetime import datetime, timedelta
from typing import Optional


# --- Mock PO Database ---
# These simulate rows from an ERP system. In production,
# you would query a real database or API instead.
MOCK_ORDERS = {
    "PO-2024-8847": {
        "status": "shipped",
        "customer": "Acme Industrial Supplies",
        "line_items": [
            {"sku": "WDG-4420", "description": "Steel Wedge Anchors (box of 100)",
             "qty": 5, "unit_price": 42.50, "status": "shipped"},
            {"sku": "BLT-1100", "description": "Hex Bolts M10x40 (box of 250)",
             "qty": 10, "unit_price": 28.00, "status": "shipped"},
        ],
        "tracking": [
            {"carrier": "FedEx", "tracking_number": "7748294001",
             "shipped_date": "2024-03-10", "eta": "2024-03-15",
             "current_location": "Memphis, TN", "status": "in_transit"}
        ],
        "total": 492.50,
        "sla_deadline": "2024-03-16",
    },
    "PO-2024-9102": {
        "status": "processing",
        "customer": "BuildRight Construction",
        "line_items": [
            {"sku": "PIP-3300", "description": "Copper Pipe 1in x 10ft",
             "qty": 20, "unit_price": 18.75, "status": "in_production"},
        ],
        "tracking": [],
        "total": 375.00,
        "sla_deadline": "2024-03-20",
    },
    "PO-2024-7500": {
        "status": "delivered",
        "customer": "Metro Office Furnishings",
        "line_items": [
            {"sku": "DSK-2200", "description": "Standing Desk Frame",
             "qty": 3, "unit_price": 289.00, "status": "delivered"},
        ],
        "tracking": [
            {"carrier": "UPS", "tracking_number": "1Z999AA10012",
             "shipped_date": "2024-03-01", "eta": "2024-03-05",
             "current_location": "Delivered", "status": "delivered"}
        ],
        "total": 867.00,
        "sla_deadline": "2024-03-06",
    },
}


def get_order_status(po_number: str) -> dict:
    """
    Simulate an ERP API call to look up order status.
    Returns order details or an error if not found.
    """
    try:
        if po_number not in MOCK_ORDERS:
            return {
                "is_error": True,
                "error_category": "not_found",
                "is_retryable": False,
                "context": f"No order found for PO number: {po_number}",
            }
        order = MOCK_ORDERS[po_number]
        return {
            "is_error": False,
            "po_number": po_number,
            "status": order["status"],
            "customer": order["customer"],
            "line_items": order["line_items"],
            "tracking": order["tracking"],
            "total": order["total"],
            "sla_deadline": order["sla_deadline"],
        }
    except Exception as e:
        return {
            "is_error": True,
            "error_category": "internal_error",
            "is_retryable": True,
            "context": str(e),
        }


def get_tracking_detail(tracking_number: str) -> dict:
    """
    Simulate a carrier tracking API call.
    In production, this would call FedEx/UPS/DHL APIs.
    """
    try:
        for order in MOCK_ORDERS.values():
            for shipment in order["tracking"]:
                if shipment["tracking_number"] == tracking_number:
                    return {
                        "is_error": False,
                        "carrier": shipment["carrier"],
                        "tracking_number": tracking_number,
                        "status": shipment["status"],
                        "current_location": shipment["current_location"],
                        "shipped_date": shipment["shipped_date"],
                        "eta": shipment["eta"],
                    }
        return {
            "is_error": True,
            "error_category": "not_found",
            "is_retryable": False,
            "context": f"No shipment found for tracking: {tracking_number}",
        }
    except Exception as e:
        return {
            "is_error": True,
            "error_category": "internal_error",
            "is_retryable": True,
            "context": str(e),
        }

// data/domain_b/loader.ts
// Domain B: B2B Ecommerce Order Tracking — Mock Data Loader
// Simulates an ERP order database and carrier tracking API
// so you can develop your agent without real API credentials.

interface LineItem {
  sku: string;
  description: string;
  qty: number;
  unitPrice: number;
  status: string;
}

interface Shipment {
  carrier: string;
  trackingNumber: string;
  shippedDate: string;
  eta: string;
  currentLocation: string;
  status: string;
}

interface Order {
  status: string;
  customer: string;
  lineItems: LineItem[];
  tracking: Shipment[];
  total: number;
  slaDeadline: string;
}

interface ToolResult {
  isError: boolean;
  errorCategory?: string;
  isRetryable?: boolean;
  context?: string;
  [key: string]: unknown;
}

// --- Mock PO Database ---
// These simulate rows from an ERP system. In production,
// you would query a real database or API instead.
const MOCK_ORDERS: Record<string, Order> = {
  "PO-2024-8847": {
    status: "shipped",
    customer: "Acme Industrial Supplies",
    lineItems: [
      { sku: "WDG-4420", description: "Steel Wedge Anchors (box of 100)",
        qty: 5, unitPrice: 42.50, status: "shipped" },
      { sku: "BLT-1100", description: "Hex Bolts M10x40 (box of 250)",
        qty: 10, unitPrice: 28.00, status: "shipped" },
    ],
    tracking: [
      { carrier: "FedEx", trackingNumber: "7748294001",
        shippedDate: "2024-03-10", eta: "2024-03-15",
        currentLocation: "Memphis, TN", status: "in_transit" }
    ],
    total: 492.50,
    slaDeadline: "2024-03-16",
  },
  "PO-2024-9102": {
    status: "processing",
    customer: "BuildRight Construction",
    lineItems: [
      { sku: "PIP-3300", description: "Copper Pipe 1in x 10ft",
        qty: 20, unitPrice: 18.75, status: "in_production" },
    ],
    tracking: [],
    total: 375.00,
    slaDeadline: "2024-03-20",
  },
  "PO-2024-7500": {
    status: "delivered",
    customer: "Metro Office Furnishings",
    lineItems: [
      { sku: "DSK-2200", description: "Standing Desk Frame",
        qty: 3, unitPrice: 289.00, status: "delivered" },
    ],
    tracking: [
      { carrier: "UPS", trackingNumber: "1Z999AA10012",
        shippedDate: "2024-03-01", eta: "2024-03-05",
        currentLocation: "Delivered", status: "delivered" }
    ],
    total: 867.00,
    slaDeadline: "2024-03-06",
  },
};


export function getOrderStatus(poNumber: string): ToolResult {
  try {
    const order = MOCK_ORDERS[poNumber];
    if (!order) {
      return {
        isError: true,
        errorCategory: "not_found",
        isRetryable: false,
        context: `No order found for PO number: ${poNumber}`,
      };
    }
    return {
      isError: false,
      poNumber,
      status: order.status,
      customer: order.customer,
      lineItems: order.lineItems,
      tracking: order.tracking,
      total: order.total,
      slaDeadline: order.slaDeadline,
    };
  } catch (error) {
    return {
      isError: true,
      errorCategory: "internal_error",
      isRetryable: true,
      context: String(error),
    };
  }
}


export function getTrackingDetail(trackingNumber: string): ToolResult {
  try {
    for (const order of Object.values(MOCK_ORDERS)) {
      for (const shipment of order.tracking) {
        if (shipment.trackingNumber === trackingNumber) {
          return {
            isError: false,
            carrier: shipment.carrier,
            trackingNumber,
            status: shipment.status,
            currentLocation: shipment.currentLocation,
            shippedDate: shipment.shippedDate,
            eta: shipment.eta,
          };
        }
      }
    }
    return {
      isError: true,
      errorCategory: "not_found",
      isRetryable: false,
      context: `No shipment found for tracking: ${trackingNumber}`,
    };
  } catch (error) {
    return {
      isError: true,
      errorCategory: "internal_error",
      isRetryable: true,
      context: String(error),
    };
  }
}

Evaluation Harness

The evaluation harness runs your agent against a set of test scenarios and scores the results automatically. Think of it as a mini test suite purpose-built for AI agents. Unlike manual testing (where you type a question, eyeball the answer, and say "looks good"), the harness runs the same scenarios every time, checks specific criteria, and gives you a pass/fail score. This means you can make a code change, rerun the harness in seconds, and immediately see if you broke something.

The harness works in three steps. First, you define test cases — each one specifies an input message, which tool the agent should (or should not) call, and what the response should contain. Second, the harness feeds each input to your agent and captures the response and tool calls. Third, it compares the actual output to the expected output and produces a score. This is the same pattern used in production agent evaluation (M18), just scaled down for a single capstone project.

Why does this matter for capstones specifically? Without a harness, you test your agent with 2-3 manual queries, declare victory, and move on. Then during self-assessment, you discover it crashes on edge cases you never tried. The harness forces you to think about edge cases upfront — before you write the code, not after. It also gives you a concrete pass rate number for your rubric's Functionality dimension.

# eval/harness.py
# Capstone Evaluation Harness
# Runs test cases against your agent and scores results.

import json
import time
from typing import Callable


# --- Test Case Format ---
# Each test case defines an input, expected behavior,
# and scoring criteria. These map to your rubric dimensions.
SAMPLE_TEST_CASES = [
    {
        "id": "TC-001",
        "description": "Valid PO number returns correct status",
        "input": "What is the status of PO-2024-8847?",
        "expected": {
            "should_call_tool": "get_order_status",
            "should_contain": ["shipped", "FedEx", "Memphis"],
            "should_not_contain": ["error", "not found"],
        },
        "rubric_dimension": "functionality",
    },
    {
        "id": "TC-002",
        "description": "Invalid PO number handled gracefully",
        "input": "What is the status of PO-9999-0000?",
        "expected": {
            "should_call_tool": "get_order_status",
            "should_contain": ["not found"],
            "should_not_contain": ["traceback", "exception", "undefined"],
        },
        "rubric_dimension": "functionality",
    },
    {
        "id": "TC-003",
        "description": "Non-order query handled appropriately",
        "input": "What is the weather today?",
        "expected": {
            "should_call_tool": None,
            "should_contain": ["order"],
            "should_not_contain": ["traceback", "exception"],
        },
        "rubric_dimension": "safety",
    },
]


def evaluate_response(
    response: str,
    tools_called: list[str],
    expected: dict,
) -> dict:
    """Score a single agent response against expectations."""
    score = {"passed": True, "details": []}

    # Check tool usage
    if expected["should_call_tool"]:
        if expected["should_call_tool"] in tools_called:
            score["details"].append("✓ Correct tool called")
        else:
            score["passed"] = False
            score["details"].append(
                f"✗ Expected tool '{expected['should_call_tool']}', "
                f"got {tools_called or 'none'}"
            )
    elif expected["should_call_tool"] is None and tools_called:
        score["passed"] = False
        score["details"].append(
            f"✗ No tool expected, but called: {tools_called}"
        )

    # Check response content
    response_lower = response.lower()
    for phrase in expected.get("should_contain", []):
        if phrase.lower() in response_lower:
            score["details"].append(f"✓ Contains '{phrase}'")
        else:
            score["passed"] = False
            score["details"].append(f"✗ Missing expected: '{phrase}'")

    for phrase in expected.get("should_not_contain", []):
        if phrase.lower() in response_lower:
            score["passed"] = False
            score["details"].append(f"✗ Contains forbidden: '{phrase}'")

    return score


def run_evaluation(
    agent_fn: Callable[[str], tuple[str, list[str]]],
    test_cases: list[dict] = SAMPLE_TEST_CASES,
) -> dict:
    """
    Run all test cases and produce a summary report.
    agent_fn should accept a user message string and return
    (response_text, list_of_tools_called).
    """
    results = []
    start = time.time()

    for tc in test_cases:
        try:
            response, tools = agent_fn(tc["input"])
            result = evaluate_response(response, tools, tc["expected"])
            result["test_id"] = tc["id"]
            result["description"] = tc["description"]
        except Exception as e:
            result = {
                "test_id": tc["id"],
                "description": tc["description"],
                "passed": False,
                "details": [f"✗ Agent crashed: {e}"],
            }
        results.append(result)

    elapsed = time.time() - start
    passed = sum(1 for r in results if r["passed"])
    total = len(results)

    return {
        "passed": passed,
        "total": total,
        "pass_rate": f"{passed/total*100:.0f}%",
        "elapsed_seconds": round(elapsed, 2),
        "results": results,
    }

// eval/harness.ts
// Capstone Evaluation Harness
// Runs test cases against your agent and scores results.

interface Expected {
  shouldCallTool: string | null;
  shouldContain: string[];
  shouldNotContain: string[];
}

interface TestCase {
  id: string;
  description: string;
  input: string;
  expected: Expected;
  rubricDimension: string;
}

interface TestResult {
  testId: string;
  description: string;
  passed: boolean;
  details: string[];
}

// --- Test Case Format ---
// Each test case defines an input, expected behavior,
// and scoring criteria. These map to your rubric dimensions.
const SAMPLE_TEST_CASES: TestCase[] = [
  {
    id: "TC-001",
    description: "Valid PO number returns correct status",
    input: "What is the status of PO-2024-8847?",
    expected: {
      shouldCallTool: "get_order_status",
      shouldContain: ["shipped", "FedEx", "Memphis"],
      shouldNotContain: ["error", "not found"],
    },
    rubricDimension: "functionality",
  },
  {
    id: "TC-002",
    description: "Invalid PO number handled gracefully",
    input: "What is the status of PO-9999-0000?",
    expected: {
      shouldCallTool: "get_order_status",
      shouldContain: ["not found"],
      shouldNotContain: ["traceback", "exception", "undefined"],
    },
    rubricDimension: "functionality",
  },
  {
    id: "TC-003",
    description: "Non-order query handled appropriately",
    input: "What is the weather today?",
    expected: {
      shouldCallTool: null,
      shouldContain: ["order"],
      shouldNotContain: ["traceback", "exception"],
    },
    rubricDimension: "safety",
  },
];


function evaluateResponse(
  response: string,
  toolsCalled: string[],
  expected: Expected,
): TestResult {
  const result: TestResult = {
    testId: "",
    description: "",
    passed: true,
    details: [],
  };

  // Check tool usage
  if (expected.shouldCallTool) {
    if (toolsCalled.includes(expected.shouldCallTool)) {
      result.details.push("✓ Correct tool called");
    } else {
      result.passed = false;
      result.details.push(
        `✗ Expected tool '${expected.shouldCallTool}', ` +
        `got ${toolsCalled.length ? toolsCalled.join(", ") : "none"}`
      );
    }
  } else if (expected.shouldCallTool === null && toolsCalled.length > 0) {
    result.passed = false;
    result.details.push(
      `✗ No tool expected, but called: ${toolsCalled.join(", ")}`
    );
  }

  // Check response content
  const lower = response.toLowerCase();
  for (const phrase of expected.shouldContain) {
    if (lower.includes(phrase.toLowerCase())) {
      result.details.push(`✓ Contains '${phrase}'`);
    } else {
      result.passed = false;
      result.details.push(`✗ Missing expected: '${phrase}'`);
    }
  }
  for (const phrase of expected.shouldNotContain) {
    if (lower.includes(phrase.toLowerCase())) {
      result.passed = false;
      result.details.push(`✗ Contains forbidden: '${phrase}'`);
    }
  }

  return result;
}


type AgentFn = (input: string) => Promise<[string, string[]]>;

export async function runEvaluation(
  agentFn: AgentFn,
  testCases: TestCase[] = SAMPLE_TEST_CASES,
) {
  const results: TestResult[] = [];
  const start = Date.now();

  for (const tc of testCases) {
    try {
      const [response, tools] = await agentFn(tc.input);
      const result = evaluateResponse(response, tools, tc.expected);
      result.testId = tc.id;
      result.description = tc.description;
      results.push(result);
    } catch (error) {
      results.push({
        testId: tc.id,
        description: tc.description,
        passed: false,
        details: [`✗ Agent crashed: ${error}`],
      });
    }
  }

  const elapsed = (Date.now() - start) / 1000;
  const passed = results.filter((r) => r.passed).length;
  const total = results.length;

  return {
    passed,
    total,
    passRate: `${Math.round((passed / total) * 100)}%`,
    elapsedSeconds: Math.round(elapsed * 100) / 100,
    results,
  };
}

Example: Capstone 1 — Minimum Viable Agent

Here is a complete "minimum viable" implementation of Capstone 1 (Domain B — Order Status Bot). This sets the bar for scope and quality. Your implementation should be at least this complete, and ideally better.

The agent uses the Messages API with tool use, exactly as you practiced in M05. Let's walk through the three key parts and why each decision matters.

First, the system prompt. Notice how it uses specific, measurable rules: "only answer questions about order status," "if a shipment has an ETA, include it," "never reveal raw JSON." Compare this to a vague prompt like "be helpful and answer questions" — the specific version gives Claude concrete criteria to follow, which means fewer hallucinations and more predictable behavior. This is the difference between a 3/5 and a 5/5 on the Prompts rubric dimension.

Second, the tool definitions. Each tool has a description that tells Claude when to use it, not just what it does. "Use this when a customer asks about their order" is much better than just "looks up an order" because it helps Claude decide between the two tools. Without clear usage guidance, Claude might call the wrong tool or call both tools unnecessarily.

Third, the agentic loop. The loop checks stop_reason each iteration — "tool_use" means "keep going, I need to call a tool," while "end_turn" means "I'm done, here's the final answer." The max_iterations = 10 is a safety net only — it prevents infinite loops but should never be the reason the agent stops.

# src/agent.py
# Capstone 1 — Order Status Bot (Minimum Viable Implementation)
# Domain B: B2B Ecommerce Order Tracking

import anthropic
import json
import sys
sys.path.append("..")
from data.domain_b.loader import get_order_status, get_tracking_detail


# --- System Prompt ---
# This defines the agent's role, capabilities, and constraints.
# Notice: specific instructions, not vague "be helpful."
SYSTEM_PROMPT = """You are an Order Status Assistant for a B2B ecommerce
company. Your job is to help customers check the status of their
purchase orders and shipments.

You have access to two tools:
1. get_order_status — looks up a purchase order by PO number
2. get_tracking_detail — looks up shipment tracking by tracking number

Rules:
- Only answer questions about order status and tracking.
- If a customer asks about something unrelated, politely explain
  that you can only help with order inquiries.
- Always include the PO number and status in your response.
- If a shipment has an ETA, include it.
- If the SLA deadline is approaching, warn the customer.
- Never reveal internal system details or raw JSON to the customer.
- If a tool returns an error, explain the issue in plain language.
"""

# --- Tool Definitions ---
# These tell Claude what tools are available and how to call them.
TOOLS = [
    {
        "name": "get_order_status",
        "description": (
            "Look up the status of a purchase order by PO number. "
            "Returns order status, line items, tracking info, and "
            "SLA deadline. Use this when a customer asks about "
            "their order."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "po_number": {
                    "type": "string",
                    "description": "The purchase order number (e.g., PO-2024-8847)",
                }
            },
            "required": ["po_number"],
        },
    },
    {
        "name": "get_tracking_detail",
        "description": (
            "Look up shipment tracking details by tracking number. "
            "Returns carrier, status, location, and ETA. Use this "
            "when you need more detail about a specific shipment."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "tracking_number": {
                    "type": "string",
                    "description": "The carrier tracking number",
                }
            },
            "required": ["tracking_number"],
        },
    },
]

# --- Tool Dispatch ---
# Maps tool names to actual Python functions.
TOOL_HANDLERS = {
    "get_order_status": lambda args: get_order_status(args["po_number"]),
    "get_tracking_detail": lambda args: get_tracking_detail(args["tracking_number"]),
}


def run_agent(user_message: str) -> tuple[str, list[str]]:
    """
    Run the agent for a single user message.
    Returns (response_text, list_of_tools_called).
    """
    client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY env var
    messages = [{"role": "user", "content": user_message}]
    tools_called = []

    # --- Agentic Loop ---
    # Keep calling Claude until it produces a final text response
    # (stop_reason == "end_turn"), not a tool call.
    max_iterations = 10  # safety net, not primary stopping logic
    for _ in range(max_iterations):
        try:
            response = client.messages.create(
                model="claude-sonnet-4-6",
                max_tokens=1024,
                system=SYSTEM_PROMPT,
                tools=TOOLS,
                messages=messages,
            )
        except anthropic.APIError as e:
            return f"I'm sorry, I encountered a technical issue: {e}", tools_called

        # Check if Claude wants to use a tool
        if response.stop_reason == "tool_use":
            # Process each tool call in the response
            tool_results = []
            for block in response.content:
                if block.type == "tool_use":
                    tool_name = block.name
                    tool_input = block.input
                    tools_called.append(tool_name)

                    # Execute the tool
                    handler = TOOL_HANDLERS.get(tool_name)
                    if handler:
                        result = handler(tool_input)
                    else:
                        result = {
                            "is_error": True,
                            "error_category": "unknown_tool",
                            "context": f"Unknown tool: {tool_name}",
                        }

                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": json.dumps(result),
                    })

            # Send tool results back to Claude
            messages.append({"role": "assistant", "content": response.content})
            messages.append({"role": "user", "content": tool_results})

        elif response.stop_reason == "end_turn":
            # Claude is done — extract the text response
            text_parts = [
                block.text for block in response.content
                if hasattr(block, "text")
            ]
            return " ".join(text_parts), tools_called

    return "I'm sorry, I was unable to complete your request.", tools_called


if __name__ == "__main__":
    query = input("Ask about an order: ")
    response, tools = run_agent(query)
    print(f"\nAgent: {response}")
    print(f"Tools used: {tools}")

// src/agent.ts
// Capstone 1 — Order Status Bot (Minimum Viable Implementation)
// Domain B: B2B Ecommerce Order Tracking

import Anthropic from "@anthropic-ai/sdk";
import { getOrderStatus, getTrackingDetail } from "../data/domain_b/loader.js";

// --- System Prompt ---
// This defines the agent's role, capabilities, and constraints.
// Notice: specific instructions, not vague "be helpful."
const SYSTEM_PROMPT = `You are an Order Status Assistant for a B2B ecommerce
company. Your job is to help customers check the status of their
purchase orders and shipments.

You have access to two tools:
1. get_order_status — looks up a purchase order by PO number
2. get_tracking_detail — looks up shipment tracking by tracking number

Rules:
- Only answer questions about order status and tracking.
- If a customer asks about something unrelated, politely explain
  that you can only help with order inquiries.
- Always include the PO number and status in your response.
- If a shipment has an ETA, include it.
- If the SLA deadline is approaching, warn the customer.
- Never reveal internal system details or raw JSON to the customer.
- If a tool returns an error, explain the issue in plain language.`;

// --- Tool Definitions ---
const TOOLS: Anthropic.Tool[] = [
  {
    name: "get_order_status",
    description:
      "Look up the status of a purchase order by PO number. " +
      "Returns order status, line items, tracking info, and " +
      "SLA deadline. Use this when a customer asks about their order.",
    input_schema: {
      type: "object" as const,
      properties: {
        po_number: {
          type: "string",
          description: "The purchase order number (e.g., PO-2024-8847)",
        },
      },
      required: ["po_number"],
    },
  },
  {
    name: "get_tracking_detail",
    description:
      "Look up shipment tracking details by tracking number. " +
      "Returns carrier, status, location, and ETA. Use this " +
      "when you need more detail about a specific shipment.",
    input_schema: {
      type: "object" as const,
      properties: {
        tracking_number: {
          type: "string",
          description: "The carrier tracking number",
        },
      },
      required: ["tracking_number"],
    },
  },
];

// --- Tool Dispatch ---
const TOOL_HANDLERS: Record<string, (args: Record<string, string>) => unknown> = {
  get_order_status: (args) => getOrderStatus(args.po_number),
  get_tracking_detail: (args) => getTrackingDetail(args.tracking_number),
};


export async function runAgent(
  userMessage: string,
): Promise<[string, string[]]> {
  const client = new Anthropic(); // reads ANTHROPIC_API_KEY env var
  const messages: Anthropic.MessageParam[] = [
    { role: "user", content: userMessage },
  ];
  const toolsCalled: string[] = [];

  // --- Agentic Loop ---
  const maxIterations = 10; // safety net
  for (let i = 0; i < maxIterations; i++) {
    let response: Anthropic.Message;
    try {
      response = await client.messages.create({
        model: "claude-sonnet-4-6",
        max_tokens: 1024,
        system: SYSTEM_PROMPT,
        tools: TOOLS,
        messages,
      });
    } catch (error) {
      return [`I'm sorry, I encountered a technical issue: ${error}`, toolsCalled];
    }

    if (response.stop_reason === "tool_use") {
      const toolResults: Anthropic.ToolResultBlockParam[] = [];

      for (const block of response.content) {
        if (block.type === "tool_use") {
          toolsCalled.push(block.name);
          const handler = TOOL_HANDLERS[block.name];
          const result = handler
            ? handler(block.input as Record<string, string>)
            : { isError: true, errorCategory: "unknown_tool",
                context: `Unknown tool: ${block.name}` };

          toolResults.push({
            type: "tool_result",
            tool_use_id: block.id,
            content: JSON.stringify(result),
          });
        }
      }

      messages.push({ role: "assistant", content: response.content });
      messages.push({ role: "user", content: toolResults });

    } else if (response.stop_reason === "end_turn") {
      const textParts = response.content
        .filter((b): b is Anthropic.TextBlock => b.type === "text")
        .map((b) => b.text);
      return [textParts.join(" "), toolsCalled];
    }
  }

  return ["I'm sorry, I was unable to complete your request.", toolsCalled];
}

Hands-On Exercise: Launch Your Capstone

What You'll Build: A working project scaffold with mock data, a running data loader, and your first agent test — everything you need to begin your capstone.

Time Estimate: 45–60 minutes

Prerequisites: Python 3.10+ or Node.js 18+ installed. An Anthropic API key (set as ANTHROPIC_API_KEY environment variable). Completed at least Tracks 1–3 (M01-M11).

Files You'll Create:

• capstone-project/ — project root directory
• data/domain_b/loader.py (or .ts) — mock data loader
• eval/test_cases.json — evaluation test scenarios
• src/agent.py (or .ts) — your agent (copied from the example above)

Environment Setup

Copy and paste the following block into your terminal to create the project and install dependencies:

# Create project structure
mkdir -p capstone-project/src capstone-project/data/domain_b capstone-project/eval capstone-project/traces
cd capstone-project

# Set up Python virtual environment
python -m venv venv
source venv/bin/activate          # On Windows: venv\Scripts\activate

# Install dependencies
pip install anthropic

# Set your API key (replace with your actual key)
export ANTHROPIC_API_KEY=your-key-here   # On Windows: set ANTHROPIC_API_KEY=your-key-here

# Create project structure
mkdir -p capstone-project/src capstone-project/data/domain_b capstone-project/eval capstone-project/traces
cd capstone-project

# Initialize Node.js project
npm init -y
npm install @anthropic-ai/sdk
npm install -D typescript @types/node
npx tsc --init

# Set your API key (replace with your actual key)
export ANTHROPIC_API_KEY=your-key-here   # On Windows: set ANTHROPIC_API_KEY=your-key-here

Step 1: Create the Mock Data Loader

What & Why: Copy the Domain B data loader from the Starter Template section above into data/domain_b/loader.py (or loader.ts). This gives your agent realistic order data to work with, without needing real API credentials.

Create: A new file at data/domain_b/loader.py containing the full data loader code shown in the "Domain Data Loader" section above.

Test it:

cd capstone-project
python -c "
from data.domain_b.loader import get_order_status, get_tracking_detail
import json

# Test valid PO
result = get_order_status('PO-2024-8847')
print('Valid PO:', json.dumps(result, indent=2)[:200])

# Test invalid PO
result = get_order_status('PO-INVALID')
print('Invalid PO:', json.dumps(result, indent=2))
"

cd capstone-project
npx tsx -e "
import { getOrderStatus } from './data/domain_b/loader.js';

const valid = getOrderStatus('PO-2024-8847');
console.log('Valid PO:', JSON.stringify(valid, null, 2).slice(0, 200));

const invalid = getOrderStatus('PO-INVALID');
console.log('Invalid PO:', JSON.stringify(invalid, null, 2));
"

Node.js note: the TypeScript loader uses camelCase keys (isError, poNumber, errorCategory, isRetryable) instead of snake_case. The values are otherwise identical.

Troubleshooting:

• ModuleNotFoundError: No module named 'data' — Create empty __init__.py files in both data/ and data/domain_b/: touch data/__init__.py data/domain_b/__init__.py
• FileNotFoundError or wrong path — Make sure you are running from the capstone-project/ root directory, not from inside data/.
• Output shows None or empty — Verify you copied the full MOCK_ORDERS dictionary and that the PO number matches exactly (PO-2024-8847, with hyphens).

Step 2: Create the Agent

What & Why: Copy the Capstone 1 agent code from the "Example: Capstone 1" section above into src/agent.py (or agent.ts). This gives you a working minimum viable agent to iterate on.

Create: A new file at src/agent.py with the full agent code shown above.

Test it:

cd capstone-project
python -c "
from src.agent import run_agent
response, tools = run_agent('What is the status of PO-2024-8847?')
print('Response:', response[:300])
print('Tools called:', tools)
"

cd capstone-project
npx tsx -e "
import { runAgent } from './src/agent.js';
const [response, tools] = await runAgent('What is the status of PO-2024-8847?');
console.log('Response:', response.slice(0, 300));
console.log('Tools called:', tools);
"

Troubleshooting:

• AuthenticationError — Your API key is missing or invalid. Run echo $ANTHROPIC_API_KEY (or echo %ANTHROPIC_API_KEY% on Windows) to verify it is set.
• ModuleNotFoundError for the data loader — The import path in agent.py (from data.domain_b.loader import ...) assumes you run from the project root. Also create src/__init__.py if needed.
• Agent returns generic "I can't help with that" — Your system prompt may be too restrictive. Check that the tools list is being passed to client.messages.create().

Step 3: Run the Evaluation Harness

What & Why: Copy the evaluation harness from the section above into eval/harness.py. Then run it against your agent to get a pass/fail score across all test cases. This gives you a concrete number for your rubric.

Create: A new file at eval/harness.py with the full harness code shown above.

Test it:

cd capstone-project
python -c "
import json
from src.agent import run_agent
from eval.harness import run_evaluation

report = run_evaluation(run_agent)
print(json.dumps(report, indent=2))
"

cd capstone-project
npx tsx -e "
import { runAgent } from './src/agent.js';
import { runEvaluation } from './eval/harness.js';
const report = await runEvaluation(runAgent);
console.log(JSON.stringify(report, null, 2));
"

Troubleshooting:

• TC-001 fails ("Missing expected: 'shipped'") — Claude's wording varies between runs. If the response says "in transit" instead of "shipped," either update the test expectation or add "in transit" as an alternative in should_contain.
• TC-002 fails ("Missing expected: 'not found'") — The agent might phrase it differently ("I couldn't locate that order"). Either update should_contain with an alternative phrase, or strengthen the system prompt to echo the tool's error message verbatim.
• TC-003 fails ("No tool expected, but called: get_order_status") — Your system prompt needs a stronger boundary. Add an explicit rule like: "If the user asks about anything other than orders or shipments, do NOT call any tools."
• ModuleNotFoundError for the harness — Create eval/__init__.py and run from the project root.

Step 4: Self-Assess with the Rubric

What & Why: Score your agent on each of the five rubric dimensions (Functionality, Code Quality, Prompts, Safety, Observability). Be honest — this is for your learning, not a grade. Write down specific notes for each dimension explaining why you gave that score.

Record your scores: Create a file called RUBRIC.md in your project root with your scores and notes, following the JSON format shown in the Self-Assessment Rubric section above.

Final Verification

Run this single command to verify your entire project scaffold is working end-to-end:

cd capstone-project && python -c "
from data.domain_b.loader import get_order_status
from src.agent import run_agent
from eval.harness import run_evaluation
print('Data loader: OK' if not get_order_status('PO-2024-8847')['is_error'] else 'FAIL')
resp, tools = run_agent('Status of PO-2024-8847?')
print('Agent: OK' if 'shipped' in resp.lower() else 'FAIL')
report = run_evaluation(run_agent)
print(f'Eval harness: {report[\"pass_rate\"]} ({report[\"passed\"]}/{report[\"total\"]})')
print('All systems go!' if report['passed'] == report['total'] else 'Some tests failed — review output above.')
"

cd capstone-project && npx tsx -e "
import { getOrderStatus } from './data/domain_b/loader.js';
import { runAgent } from './src/agent.js';
import { runEvaluation } from './eval/harness.js';
console.log('Data loader:', !getOrderStatus('PO-2024-8847').isError ? 'OK' : 'FAIL');
const [resp, tools] = await runAgent('Status of PO-2024-8847?');
console.log('Agent:', resp.toLowerCase().includes('shipped') ? 'OK' : 'FAIL');
const report = await runEvaluation(runAgent);
console.log('Eval harness:', report.passRate, '(' + report.passed + '/' + report.total + ')');
console.log(report.passed === report.total ? 'All systems go!' : 'Some tests failed.');
"

Troubleshooting

• ModuleNotFoundError: No module named 'anthropic' — Run pip install anthropic (make sure your virtual environment is activated).
• AuthenticationError — Check that ANTHROPIC_API_KEY is set: run echo $ANTHROPIC_API_KEY (or echo %ANTHROPIC_API_KEY% on Windows).
• ModuleNotFoundError: No module named 'data' — Create empty __init__.py files in data/ and data/domain_b/. Run from the project root directory.
• Agent response does not mention "shipped" — Claude's exact wording varies. Check the raw response. If it says "in transit" instead of "shipped," update your test expectations or your system prompt.

Stretch goal (OPTIONAL): Complete a second capstone in a different domain to practice adapting your skills across industries.

Stretch goal (OPTIONAL): Implement the "extended" version with full production hardening (observability, caching, deployment from M19-M22).

Knowledge Check

Module Summary

In this module, you learned the complete framework for tackling capstone projects:

• Five-Phase Methodology: Requirements → Architecture → Build → Evaluate → Harden. Each phase maps to specific course modules, preventing the "patch-and-pray" cycle.
• Three Difficulty Tiers: Beginner (Capstone 1), Intermediate (Capstones 2–3), and Advanced (Capstones 4–5). Pick the tier that matches the tracks you have completed.
• Three Domain Anchors: Healthcare (HIPAA, clinical criteria), B2B Ecommerce (PO lifecycle, carrier APIs), and Public Records (UCC filings, entity resolution). Same agent patterns, different real-world contexts.
• Self-Assessment Rubric: Five dimensions (Functionality, Code Quality, Prompts, Safety, Observability) scored 1–5. 15+ = competent, 20+ = excellent.
• Starter Template + Eval Harness: Directory structure, domain data loaders, and automated test runner — everything you need to start building immediately.

Next up: M24: What's Next — The Agent Frontier explores emerging patterns (agent-to-agent protocols, agent marketplaces), Claude's evolving capabilities (computer use, extended thinking), and your personal agent development roadmap.

References & Resources

• Claude Tool Use Documentation — Official guide to function calling and tool use
• Claude Model Overview — Model capabilities, pricing, and selection guidance
• Anthropic Cookbook — Production-ready code examples for agents, RAG, and tool use
• Prompt Engineering Guide — Best practices for system prompts and structured output
• Evaluation Guide — Anthropic's guidance on testing and evaluating AI outputs