⌂ Home MCP Track — Building with the Model Context Protocol ⚡ MODULE 07 of 8 · MCP Track
~75 min Intermediate → Advanced Lab — 70% Code
MODULE 07 · Multi-Server Orchestration

Multi-Server Orchestration

Connect multiple specialized MCP servers into a single coherent agent. Learn topology patterns, tool namespacing, smart routing, load balancing, and distributed observability — all with complete Python and TypeScript implementations.

After this module you will:
  • Explain the three topology patterns (fan-out, pipeline, hierarchical) and when to use each
  • Implement prefix-based and schema-based tool namespacing to prevent collisions
  • Build a SmartRouter with static map, capability match, and intent routing strategies
  • Implement load balancing with round-robin, least-connections, and automatic failover
  • Add distributed tracing to reconstruct cross-server call graphs

1. Why Orchestrate Multiple Servers?

📚 Analogy — The Specialist Consultant

BEFORE: Imagine you hire one expert consultant for a complex project. They are world-class at financial modeling — but when you need a marketing strategy or a legal opinion, they shrug. You hired a specialist, not a generalist.

PAIN: A single MCP server is exactly that specialist. It exposes the tools of one domain perfectly — your database server handles every SQL query with precision — but the moment your agent needs to score a risk metric, store a result, or fire a Slack notification, that server has nothing to offer. The LLM sees a narrow toolset and is forced to either hallucinate or give up.

MAPPING: Orchestrating multiple MCP servers gives your agent a full consulting team: one specialist for data, one for compute, one for storage, one for communications. The orchestrating host surfaces all their tools simultaneously, and the LLM picks the right expert for each sub-task — just as a project manager routes work to the right consultant without asking the client to manage the logistics.

In practice, a production agent almost always needs at least four capability domains:

Domain Typical Server Example Tools
Data DataServer search_filings, get_filing_details, run_sql
Compute ComputeServer fuzzy_match_score, calculate_risk, ml_classify
Storage StorageServer save_result, recall_similar, delete_record
Communications CommsServer send_email, post_slack, create_ticket
📈 Why It Matters

Teams at Anthropic building production agents report that a typical enterprise workflow touches 3–5 MCP servers per agent run. Without orchestration, each domain becomes a separate agent with its own context window — multiplying latency and cost. A single well-orchestrated host cuts round-trips by 60-80% compared to chaining separate agent calls.

📚 Technical Definition — MCP Orchestration

MCP orchestration means a single MCP hostMCP Host — the process that manages connections to one or more MCP servers, aggregates their tool lists, and routes tool calls. Claude Desktop and Claude Code are hosts; you can also build a custom host in Python or TypeScript. maintains simultaneous connections to N servers, merges their tool schemas into one list that the LLM sees, and dispatches each tool call to the correct server — all transparently to the model.

2. Topology Patterns

There are three canonical ways to wire multiple servers together. Choosing the wrong topology for your workload is one of the most common architectural mistakes — each has clear trade-offs you should understand before writing a line of code.

Animation 1 — Three Topology Patterns

Fan-out: All Tools Visible Simultaneously

The host connects to all N servers on startup and merges their tool lists. The LLM sees every tool in a single list and picks freely. Best for: general-purpose agents where you cannot predict which capabilities will be needed per turn.

  • Trade-off: Large tool lists (30+ tools) slightly increase prompt token count and can confuse smaller models. Mitigate with intent routing (Section 4).

Pipeline: Chained Processing

Server A's output becomes Server B's input. The host enforces a processing order — it calls Server A, passes the structured result to Server B, and returns the final output. Best for: ETL-style workflows where step N always feeds step N+1.

  • Trade-off: Inflexible. If the agent decides mid-run that it needs to skip a stage, the pipeline fights you. Reserve for truly sequential transformations.

Hierarchical: Supervisor + Sub-Servers

A supervisor serverSupervisor Server — an MCP server that is itself an MCP client. It exposes high-level abstract tools to the LLM (e.g., analyze_entity), but internally calls sub-servers to fulfill them. The LLM never sees the sub-server complexity. acts as a client to a set of sub-servers. It exposes only high-level abstract tools to the host LLM. Best for: capability abstraction, where you want to present a simplified API to the model and hide internal orchestration logic.

  • Trade-off: Adds a network hop. The supervisor must be a full MCP server+client simultaneously — more code to maintain. Worth it when tool list noise is a real problem.
PatternTool VisibilityBest ForMain Risk
Fan-outAll tools mergedGeneral agentsTool list bloat
PipelineSequential exposureETL / transformsInflexibility
HierarchicalAbstract facadeCapability abstractionExtra hop latency
You now know the shape of multi-server systems. Next comes the practical plumbing problem: what happens when two servers both expose a tool called search? Tool namespacing solves this before it ever reaches the router.

3. Tool Namespacing

When two servers both expose search, the host has a collision. Without disambiguation, the first server registered wins — silently. Tool namespacingTool namespacing — a strategy for making tool names globally unique across multiple servers by prepending a server identifier. Prevents silent collision where the wrong server handles a tool call. makes every tool name globally unique so the router always calls the right server.

Strategy 1: Prefix-based Namespacing

The host rewrites tool names on ingestion: search from the DB server becomes db::search; search from the ML server becomes ml::search. The LLM sees prefixed names; the host strips the prefix before forwarding the call.

WHAT
MCPHost.add_server() registers a server with an alias. _build_namespaced_tools() rewrites the tool list. call_tool() parses the :: separator and routes to the right client.
WHY
Prefix namespacing is zero-overhead: the LLM explicitly states the server in every tool call, eliminating any ambiguity at routing time. No inference required.
Python — prefix namespacing
"""
Prefix-based tool namespacing: db::search, ml::search
Each server gets an alias; the host rewrites tool names on registration.
"""

import asyncio
import json
from contextlib import AsyncExitStack
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from mcp.types import Tool


class MCPHost:
    """
    WHAT: Manages N server connections; rewrites tool names with server prefixes.
    WHY:  Prevents silent collisions when two servers expose the same tool name.
    """

    def __init__(self):
        self._sessions: dict[str, ClientSession] = {}
        self._namespaced_tools: list[dict] = []  # merged list sent to LLM
        self._tool_to_server: dict[str, str] = {}  # namespaced_name → alias
        self._exit_stack = AsyncExitStack()

    async def add_server(
        self,
        alias: str,                # e.g. "db", "ml", "storage"
        command: str,              # e.g. "python"
        args: list[str],           # e.g. ["data_server.py"]
        env: dict | None = None,
    ) -> None:
        """
        WHAT: Connect to a stdio MCP server and register its tools under alias::.
        GOTCHA: alias must be unique; duplicate aliases overwrite the previous entry.
        """
        params = StdioServerParameters(command=command, args=args, env=env)
        read, write = await self._exit_stack.enter_async_context(
            stdio_client(params)
        )
        session = await self._exit_stack.enter_async_context(
            ClientSession(read, write)
        )
        await session.initialize()
        self._sessions[alias] = session

        # Fetch tools and namespace them
        tools_result = await session.list_tools()
        for tool in tools_result.tools:
            namespaced_name = f"{alias}::{tool.name}"
            self._namespaced_tools.append({
                "name": namespaced_name,
                "description": f"[{alias}] {tool.description}",
                "input_schema": tool.inputSchema,
            })
            self._tool_to_server[namespaced_name] = alias

    def get_tools_for_llm(self) -> list[dict]:
        """Return the full namespaced tool list to pass to the Anthropic API."""
        return [
            {
                "name": t["name"],
                "description": t["description"],
                "input_schema": t["input_schema"],
            }
            for t in self._namespaced_tools
        ]

    async def call_tool(self, namespaced_name: str, tool_input: dict) -> str:
        """
        WHAT: Parse alias::tool_name, look up the right session, forward the call.
        WHY:  The LLM always uses the namespaced name; this is the single routing point.
        GOTCHA: If alias is unknown, raise immediately — never silently drop calls.
        """
        if "::" not in namespaced_name:
            raise ValueError(
                f"Tool name '{namespaced_name}' missing namespace prefix (expected alias::tool_name)"
            )
        alias, original_name = namespaced_name.split("::", 1)
        if alias not in self._sessions:
            raise KeyError(f"No server registered with alias '{alias}'")
        session = self._sessions[alias]
        result = await session.call_tool(original_name, tool_input)
        # Extract text content from MCP result
        if result.content:
            return result.content[0].text if hasattr(result.content[0], "text") else str(result.content[0])
        return ""

    async def close(self) -> None:
        await self._exit_stack.aclose()


# ── Usage ────────────────────────────────────────────────────────────────────
async def demo():
    host = MCPHost()
    try:
        await host.add_server("db", "python", ["data_server.py"])
        await host.add_server("ml", "python", ["compute_server.py"])
        await host.add_server("store", "python", ["storage_server.py"])

        tools = host.get_tools_for_llm()
        # Tools are now: db::search_filings, ml::fuzzy_match_score, store::save_result, …
        for t in tools:
            print(f"  {t['name']}: {t['description'][:60]}")

        # Route a call — alias extracted automatically
        result = await host.call_tool("db::search_filings", {"query": "Acme Corp"})
        print("Result:", result)
    finally:
        await host.close()

asyncio.run(demo())
TypeScript — prefix namespacing
/**
 * Prefix-based tool namespacing: db::search, ml::search
 * WHAT: MCPHost manages N stdio servers and rewrites tool names.
 * WHY:  The LLM always emits alias::toolName, so routing is O(1).
 */

import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";
import { Tool } from "@modelcontextprotocol/sdk/types.js";

interface ServerEntry {
  client: Client;
  transport: StdioClientTransport;
}

interface NamespacedTool {
  name: string;
  description: string;
  input_schema: Record;
}

export class MCPHost {
  private sessions = new Map<string, ServerEntry>();
  private namespacedTools: NamespacedTool[] = [];
  private toolToServer = new Map<string, string>(); // namespaced → alias

  async addServer(
    alias: string,
    command: string,
    args: string[],
    env?: Record<string, string>
  ): Promise<void> {
    const transport = new StdioClientTransport({ command, args, env });
    const client = new Client({ name: "mcp-host", version: "1.0.0" }, {});
    await client.connect(transport);
    this.sessions.set(alias, { client, transport });

    // Fetch and namespace tools
    const { tools } = await client.listTools();
    for (const tool of tools) {
      const namespacedName = `${alias}::${tool.name}`;
      this.namespacedTools.push({
        name: namespacedName,
        description: `[${alias}] ${tool.description ?? ""}`,
        input_schema: (tool.inputSchema as Record<string, unknown>) ?? {},
      });
      this.toolToServer.set(namespacedName, alias);
    }
  }

  getToolsForLLM(): NamespacedTool[] {
    return [...this.namespacedTools];
  }

  async callTool(
    namespacedName: string,
    toolInput: Record<string, unknown>
  ): Promise<string> {
    if (!namespacedName.includes("::")) {
      throw new Error(
        `Tool '${namespacedName}' missing namespace prefix (expected alias::toolName)`
      );
    }
    const [alias, originalName] = namespacedName.split("::", 2);
    const entry = this.sessions.get(alias);
    if (!entry) throw new Error(`No server registered with alias '${alias}'`);

    const result = await entry.client.callTool({
      name: originalName,
      arguments: toolInput,
    });
    const content = result.content as Array<{ type: string; text?: string }>;
    return content.find((c) => c.type === "text")?.text ?? "";
  }

  async close(): Promise<void> {
    for (const { client, transport } of this.sessions.values()) {
      await client.close();
      await transport.close();
    }
  }
}

// ── Usage ─────────────────────────────────────────────────────────────────
const host = new MCPHost();
await host.addServer("db", "node", ["data_server.js"]);
await host.addServer("ml", "node", ["compute_server.js"]);
await host.addServer("store", "node", ["storage_server.js"]);

const tools = host.getToolsForLLM();
// → db::search_filings, ml::fuzzy_match_score, store::save_result …
console.log(tools.map((t) => t.name));

const result = await host.callTool("db::search_filings", { query: "Acme Corp" });
console.log("Result:", result);
await host.close();
What Just Happened?

When add_server("db", ...) runs, it fetches the raw tool list from the DB server (search_filings, get_filing_details) and rewrites each name to db::search_filings, db::get_filing_details. When the LLM later emits db::search_filings, the host splits on ::, finds the db session, and calls search_filings on it — the server never sees the prefix.

Strategy 2: Schema-based Namespacing

Instead of rewriting names, include a server field in the tool's input_schema metadata. The router reads this field from the LLM's tool call to determine destination. Less common — use prefix-based unless you have a specific reason to keep original tool names.

⚠ Gotcha

Schema-based namespacing requires the LLM to correctly fill in the server field on every call. Models occasionally omit optional fields. Prefix-based puts the routing info in the name, which the model always provides.

4. Routing Strategies

The host must decide which server to call. The choice of routing strategy affects latency, reliability, and tool list size. Three strategies cover virtually every real use case.

Animation 2 — Tool Routing Decision
WHAT
SmartRouter wraps the MCPHost and selects a routing strategy at call time. Three strategies: static (dict lookup), capability (match tool name in per-server list), and intent (run a fast LLM classification first).
WHY
Static routing is fastest (nanosecond dict lookup) but requires a pre-built map. Capability matching handles dynamic tool registration without configuration. Intent routing shrinks the tool list seen by the primary model, which reduces hallucination on complex agents with 40+ tools.
GOTCHA
Intent routing adds a second LLM call per turn. Use haiku for the classifier — it costs ~200x less than Sonnet and classification accuracy is equivalent.
Python — SmartRouter (all three strategies)
"""
SmartRouter — three configurable routing strategies.
WHAT: Wraps MCPHost and selects server-specific tool subsets based on strategy.
WHY:  Reduces tool list noise, improves routing accuracy, lowers token cost.
"""

import anthropic
from enum import Enum
from typing import Literal


class RoutingStrategy(str, Enum):
    STATIC = "static"          # fastest: pre-built name→server dict
    CAPABILITY = "capability"  # flexible: match tool name in per-server list
    INTENT = "intent"          # smartest: classify intent, show only relevant tools


class SmartRouter:
    def __init__(
        self,
        host: "MCPHost",                             # from Section 3
        strategy: RoutingStrategy = RoutingStrategy.CAPABILITY,
        anthropic_client: anthropic.Anthropic | None = None,
    ):
        self.host = host
        self.strategy = strategy
        self._client = anthropic_client or anthropic.Anthropic()
        # STATIC strategy: pre-built map of tool_name → server_alias
        self._static_map: dict[str, str] = {}
        # Per-server tool lists for CAPABILITY strategy
        self._server_tools: dict[str, list[str]] = {}

    def configure_static(self, tool_to_server: dict[str, str]) -> None:
        """Register explicit tool→server mappings for STATIC strategy."""
        self._static_map = tool_to_server

    def configure_server_tools(self, server_tools: dict[str, list[str]]) -> None:
        """
        Register per-server tool names for CAPABILITY strategy.
        e.g. {"db": ["search_filings", "get_filing_details"], "ml": ["fuzzy_match_score"]}
        """
        self._server_tools = server_tools

    def _route_static(self, tool_name: str) -> str:
        """O(1) dict lookup — fastest possible routing."""
        # Strip namespace prefix if present
        base = tool_name.split("::")[-1] if "::" in tool_name else tool_name
        server = self._static_map.get(base)
        if server is None:
            raise KeyError(f"No static route for tool '{tool_name}'")
        return server

    def _route_capability(self, tool_name: str) -> str:
        """Iterate per-server tool lists; return first match."""
        base = tool_name.split("::")[-1] if "::" in tool_name else tool_name
        for server_alias, tools in self._server_tools.items():
            if base in tools:
                return server_alias
        raise KeyError(f"No capability match for tool '{tool_name}'")

    async def _classify_intent(self, user_message: str) -> list[str]:
        """
        Use a cheap haiku model to classify which servers are relevant.
        Returns a list of server aliases, e.g. ["db", "ml"].
        WHY: Reduces the tool list shown to the primary model from N to a subset.
        """
        server_list = ", ".join(self._server_tools.keys())
        prompt = (
            f"Which of these MCP servers are needed to answer this request: {server_list}?\n"
            f"Request: {user_message}\n"
            "Respond with ONLY a JSON array of server names, e.g. [\"db\", \"ml\"]."
        )
        resp = self._client.messages.create(
            model="claude-haiku-4-5",
            max_tokens=64,
            messages=[{"role": "user", "content": prompt}],
        )
        import json
        text = resp.content[0].text.strip()
        # Safely parse; fall back to all servers on parse error
        try:
            servers = json.loads(text)
            return [s for s in servers if s in self._server_tools]
        except (json.JSONDecodeError, TypeError):
            return list(self._server_tools.keys())

    def get_tools_for_strategy(self, servers: list[str] | None = None) -> list[dict]:
        """
        Return the tool subset for the given servers (intent strategy).
        If servers is None, return full list (static/capability strategies).
        """
        if servers is None:
            return self.host.get_tools_for_llm()
        all_tools = self.host.get_tools_for_llm()
        return [
            t for t in all_tools
            if t["name"].split("::")[0] in servers
        ]

    async def run_agent(
        self,
        user_message: str,
        system_prompt: str = "You are a helpful research agent.",
        max_turns: int = 10,
    ) -> str:
        """
        WHAT: Full agent loop using the configured routing strategy.
        WHY:  Single entry point — caller doesn't need to know which strategy is active.
        """
        # Determine tool list based on strategy
        if self.strategy == RoutingStrategy.INTENT:
            relevant_servers = await self._classify_intent(user_message)
            tools = self.get_tools_for_strategy(relevant_servers)
        else:
            tools = self.get_tools_for_strategy()

        messages = [{"role": "user", "content": user_message}]

        for _ in range(max_turns):
            resp = self._client.messages.create(
                model="claude-sonnet-4-5",
                max_tokens=4096,
                system=system_prompt,
                tools=tools,
                messages=messages,
            )

            if resp.stop_reason == "end_turn":
                # Extract final text
                for block in resp.content:
                    if hasattr(block, "text"):
                        return block.text
                return ""

            if resp.stop_reason != "tool_use":
                break

            # Process all tool calls in this turn
            messages.append({"role": "assistant", "content": resp.content})
            tool_results = []

            for block in resp.content:
                if block.type != "tool_use":
                    continue
                tool_name = block.name  # already namespaced: db::search_filings

                # Route based on strategy
                if self.strategy == RoutingStrategy.STATIC:
                    alias = self._route_static(tool_name)
                    call_name = f"{alias}::{tool_name}" if "::" not in tool_name else tool_name
                elif self.strategy == RoutingStrategy.CAPABILITY:
                    alias = self._route_capability(tool_name)
                    call_name = f"{alias}::{tool_name}" if "::" not in tool_name else tool_name
                else:
                    # INTENT and prefix-based: tool_name is already namespaced
                    call_name = tool_name

                try:
                    result = await self.host.call_tool(call_name, block.input)
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": result,
                    })
                except Exception as e:
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": f"Error: {e}",
                        "is_error": True,
                    })

            messages.append({"role": "user", "content": tool_results})

        return "Max turns reached without final answer."


# ── Usage ─────────────────────────────────────────────────────────────────
async def main():
    host = MCPHost()
    await host.add_server("db", "python", ["data_server.py"])
    await host.add_server("ml", "python", ["compute_server.py"])
    await host.add_server("store", "python", ["storage_server.py"])

    router = SmartRouter(host, strategy=RoutingStrategy.INTENT)
    router.configure_server_tools({
        "db": ["search_filings", "get_filing_details"],
        "ml": ["fuzzy_match_score", "calculate_risk_score"],
        "store": ["save_result", "recall_similar"],
    })

    answer = await router.run_agent(
        "Find all filings for Acme Corp and calculate their risk score."
    )
    print(answer)
    await host.close()

asyncio.run(main())
TypeScript — SmartRouter (all three strategies)
/**
 * SmartRouter — three configurable routing strategies.
 * WHAT: Wraps MCPHost and selects server-specific tool subsets.
 * WHY:  Reduces tool list noise, improves routing accuracy.
 */

import Anthropic from "@anthropic-ai/sdk";
import type { MCPHost } from "./mcp-host.js";

type RoutingStrategy = "static" | "capability" | "intent";

export class SmartRouter {
  private staticMap = new Map<string, string>();     // tool → alias
  private serverTools = new Map<string, string[]>();  // alias → tools

  constructor(
    private host: MCPHost,
    private strategy: RoutingStrategy = "capability",
    private client = new Anthropic()
  ) {}

  configureStatic(map: Record<string, string>): void {
    Object.entries(map).forEach(([k, v]) => this.staticMap.set(k, v));
  }

  configureServerTools(tools: Record<string, string[]>): void {
    Object.entries(tools).forEach(([k, v]) => this.serverTools.set(k, v));
  }

  private routeStatic(toolName: string): string {
    const base = toolName.includes("::") ? toolName.split("::")[1] : toolName;
    const server = this.staticMap.get(base);
    if (!server) throw new Error(`No static route for '${toolName}'`);
    return server;
  }

  private routeCapability(toolName: string): string {
    const base = toolName.includes("::") ? toolName.split("::")[1] : toolName;
    for (const [alias, tools] of this.serverTools) {
      if (tools.includes(base)) return alias;
    }
    throw new Error(`No capability match for '${toolName}'`);
  }

  private async classifyIntent(message: string): Promise<string[]> {
    const servers = [...this.serverTools.keys()].join(", ");
    const resp = await this.client.messages.create({
      model: "claude-haiku-4-5",
      max_tokens: 64,
      messages: [
        {
          role: "user",
          content:
            `Which of these MCP servers are needed: ${servers}?\n` +
            `Request: ${message}\n` +
            'Respond with ONLY a JSON array, e.g. ["db","ml"].',
        },
      ],
    });
    const text = (resp.content[0] as { text: string }).text.trim();
    try {
      const parsed = JSON.parse(text) as string[];
      return parsed.filter((s) => this.serverTools.has(s));
    } catch {
      return [...this.serverTools.keys()];
    }
  }

  private getToolsForServers(servers?: string[]): Anthropic.Tool[] {
    const all = this.host.getToolsForLLM();
    if (!servers) return all as Anthropic.Tool[];
    return all.filter((t) =>
      servers.includes(t.name.split("::")[0])
    ) as Anthropic.Tool[];
  }

  async runAgent(
    userMessage: string,
    systemPrompt = "You are a helpful research agent.",
    maxTurns = 10
  ): Promise<string> {
    let tools: Anthropic.Tool[];
    if (this.strategy === "intent") {
      const relevant = await this.classifyIntent(userMessage);
      tools = this.getToolsForServers(relevant);
    } else {
      tools = this.getToolsForServers();
    }

    const messages: Anthropic.MessageParam[] = [
      { role: "user", content: userMessage },
    ];

    for (let i = 0; i < maxTurns; i++) {
      const resp = await this.client.messages.create({
        model: "claude-sonnet-4-5",
        max_tokens: 4096,
        system: systemPrompt,
        tools,
        messages,
      });

      if (resp.stop_reason === "end_turn") {
        const textBlock = resp.content.find((b) => b.type === "text") as
          | { type: "text"; text: string }
          | undefined;
        return textBlock?.text ?? "";
      }
      if (resp.stop_reason !== "tool_use") break;

      messages.push({ role: "assistant", content: resp.content });
      const toolResults: Anthropic.ToolResultBlockParam[] = [];

      for (const block of resp.content) {
        if (block.type !== "tool_use") continue;
        const toolName = block.name;

        let callName = toolName;
        if (this.strategy === "static") {
          const alias = this.routeStatic(toolName);
          callName = toolName.includes("::") ? toolName : `${alias}::${toolName}`;
        } else if (this.strategy === "capability") {
          const alias = this.routeCapability(toolName);
          callName = toolName.includes("::") ? toolName : `${alias}::${toolName}`;
        }

        try {
          const result = await this.host.callTool(
            callName,
            block.input as Record<string, unknown>
          );
          toolResults.push({ type: "tool_result", tool_use_id: block.id, content: result });
        } catch (e) {
          toolResults.push({
            type: "tool_result",
            tool_use_id: block.id,
            content: `Error: ${e}`,
            is_error: true,
          });
        }
      }
      messages.push({ role: "user", content: toolResults });
    }
    return "Max turns reached.";
  }
}

// ── Usage ─────────────────────────────────────────────────────────────────
const host = new MCPHost();
await host.addServer("db", "node", ["data_server.js"]);
await host.addServer("ml", "node", ["compute_server.js"]);

const router = new SmartRouter(host, "intent");
router.configureServerTools({
  db: ["search_filings", "get_filing_details"],
  ml: ["fuzzy_match_score", "calculate_risk_score"],
});

const answer = await router.runAgent(
  "Find Acme Corp filings and score their risk."
);
console.log(answer);
await host.close();
✓ Checkpoint

You have three routing strategies: (1) static map — use when tool→server assignments are fixed and you want zero overhead; (2) capability match — use when servers register tools dynamically at startup; (3) intent routing — use when tool list size causes model confusion (typically 30+ tools) and you can afford a fast classifier call.

5. Load Balancing & Failover

Production agents need high availability. A single server instance is a single point of failure — one crash, one deployment, or one network blip takes your agent offline. Load balancingLoad balancing — distributing incoming tool calls across multiple instances of the same server so no single instance is overwhelmed and a failing instance can be bypassed automatically. solves both availability and throughput.

Animation 3 — Load Balancing & Failover
WHAT
LoadBalancedMCPPool maintains a pool of sessions for the same logical server. Round-robin distributes calls evenly; least-connections routes to the session with fewest in-flight calls. A background health-check loop pings every session every 30 s.
WHY
Round-robin is stateless and fast. Least-connections is smarter for servers with variable-latency tools (e.g., a server that does ML inference). Health checks catch silent failures before they cascade into agent errors.
Python — LoadBalancedMCPPool
"""
LoadBalancedMCPPool — multiple instances of the same logical server.
WHAT: Round-robin or least-connections across N replicas; auto-failover.
WHY:  Zero-downtime deployments, high-throughput parallelism, HA.
"""

import asyncio
import time
from contextlib import AsyncExitStack
from dataclasses import dataclass, field
from enum import Enum
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client


class LBStrategy(str, Enum):
    ROUND_ROBIN = "round_robin"
    LEAST_CONNECTIONS = "least_connections"


@dataclass
class PoolEntry:
    session: ClientSession
    alias: str
    healthy: bool = True
    in_flight: int = 0
    last_check: float = field(default_factory=time.time)


class LoadBalancedMCPPool:
    """
    Pool of identical MCP server instances for HA and throughput.
    GOTCHA: All instances must expose identical tool schemas. If they diverge
    after a partial upgrade, the LLM may call a tool that doesn't exist on
    the routed instance. Use versioned deployments to prevent this.
    """

    HEALTH_CHECK_INTERVAL = 30  # seconds

    def __init__(self, strategy: LBStrategy = LBStrategy.ROUND_ROBIN):
        self.strategy = strategy
        self._entries: list[PoolEntry] = []
        self._rr_index: int = 0
        self._exit_stack = AsyncExitStack()
        self._health_task: asyncio.Task | None = None

    async def add_instance(
        self, command: str, args: list[str], env: dict | None = None, alias: str | None = None
    ) -> None:
        """Connect a new server instance to the pool."""
        params = StdioServerParameters(command=command, args=args, env=env)
        read, write = await self._exit_stack.enter_async_context(stdio_client(params))
        session = await self._exit_stack.enter_async_context(ClientSession(read, write))
        await session.initialize()
        idx = len(self._entries)
        self._entries.append(PoolEntry(
            session=session,
            alias=alias or f"instance-{idx}",
        ))

    async def start_health_checks(self) -> None:
        """Launch background health-check loop."""
        self._health_task = asyncio.create_task(self._health_loop())

    async def _health_loop(self) -> None:
        """Ping each instance every 30 s; mark unhealthy on failure."""
        while True:
            await asyncio.sleep(self.HEALTH_CHECK_INTERVAL)
            for entry in self._entries:
                try:
                    # list_tools() is a lightweight ping — no side effects
                    await asyncio.wait_for(entry.session.list_tools(), timeout=5.0)
                    entry.healthy = True
                    entry.last_check = time.time()
                except Exception as e:
                    entry.healthy = False
                    print(f"[HealthCheck] {entry.alias} UNHEALTHY: {e}")

    def _pick_round_robin(self) -> PoolEntry:
        """
        WHAT: Advance index, skip unhealthy, wrap around.
        GOTCHA: If all instances are unhealthy, raise immediately — never loop forever.
        """
        healthy = [e for e in self._entries if e.healthy]
        if not healthy:
            raise RuntimeError("All pool instances are unhealthy — no call possible")
        self._rr_index = (self._rr_index + 1) % len(healthy)
        return healthy[self._rr_index % len(healthy)]

    def _pick_least_connections(self) -> PoolEntry:
        """Pick the healthy instance with the fewest in-flight calls."""
        healthy = [e for e in self._entries if e.healthy]
        if not healthy:
            raise RuntimeError("All pool instances are unhealthy")
        return min(healthy, key=lambda e: e.in_flight)

    def _pick(self) -> PoolEntry:
        if self.strategy == LBStrategy.ROUND_ROBIN:
            return self._pick_round_robin()
        return self._pick_least_connections()

    async def call_tool(self, tool_name: str, tool_input: dict) -> str:
        """
        WHAT: Pick an instance, increment in-flight counter, call, decrement.
        WHY:  in_flight tracking enables least-connections strategy.
        GOTCHA: Always decrement in-flight even on exception (use try/finally).
        """
        entry = self._pick()
        entry.in_flight += 1
        try:
            result = await entry.session.call_tool(tool_name, tool_input)
            content = result.content
            if content:
                return content[0].text if hasattr(content[0], "text") else str(content[0])
            return ""
        except Exception:
            # Mark instance unhealthy on tool call failure
            entry.healthy = False
            raise
        finally:
            entry.in_flight = max(0, entry.in_flight - 1)

    async def close(self) -> None:
        if self._health_task:
            self._health_task.cancel()
        await self._exit_stack.aclose()


# ── Usage ─────────────────────────────────────────────────────────────────
async def main():
    pool = LoadBalancedMCPPool(strategy=LBStrategy.LEAST_CONNECTIONS)

    # Start 3 replicas of the same compute server
    for port_suffix in range(3):
        await pool.add_instance(
            "python", ["compute_server.py"],
            alias=f"compute-{port_suffix}",
        )

    await pool.start_health_checks()

    # Calls are automatically distributed across healthy instances
    results = await asyncio.gather(
        pool.call_tool("fuzzy_match_score", {"a": "Acme Inc", "b": "Acme Corp"}),
        pool.call_tool("calculate_risk_score", {"entity_id": "E001"}),
        pool.call_tool("fuzzy_match_score", {"a": "GlobalTech", "b": "Global Technologies"}),
    )
    for r in results:
        print(r)

    await pool.close()

asyncio.run(main())
TypeScript — LoadBalancedMCPPool
/**
 * LoadBalancedMCPPool — multiple instances of the same logical server.
 * WHAT: Round-robin or least-connections; health-check loop; auto-failover.
 */

import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";

type LBStrategy = "round_robin" | "least_connections";

interface PoolEntry {
  client: Client;
  transport: StdioClientTransport;
  alias: string;
  healthy: boolean;
  inFlight: number;
  lastCheck: number;
}

export class LoadBalancedMCPPool {
  private entries: PoolEntry[] = [];
  private rrIndex = 0;
  private healthInterval?: ReturnType<typeof setInterval>;
  private readonly HEALTH_INTERVAL_MS = 30_000;

  constructor(private strategy: LBStrategy = "round_robin") {}

  async addInstance(
    command: string,
    args: string[],
    alias?: string,
    env?: Record<string, string>
  ): Promise<void> {
    const transport = new StdioClientTransport({ command, args, env });
    const client = new Client({ name: "lb-pool", version: "1.0.0" }, {});
    await client.connect(transport);
    this.entries.push({
      client,
      transport,
      alias: alias ?? `instance-${this.entries.length}`,
      healthy: true,
      inFlight: 0,
      lastCheck: Date.now(),
    });
  }

  startHealthChecks(): void {
    this.healthInterval = setInterval(async () => {
      for (const entry of this.entries) {
        try {
          await Promise.race([
            entry.client.listTools(),
            new Promise((_, rej) => setTimeout(() => rej(new Error("timeout")), 5000)),
          ]);
          entry.healthy = true;
          entry.lastCheck = Date.now();
        } catch (e) {
          entry.healthy = false;
          console.warn(`[HealthCheck] ${entry.alias} UNHEALTHY: ${e}`);
        }
      }
    }, this.HEALTH_INTERVAL_MS);
  }

  private pickRoundRobin(): PoolEntry {
    const healthy = this.entries.filter((e) => e.healthy);
    if (!healthy.length) throw new Error("All pool instances are unhealthy");
    this.rrIndex = (this.rrIndex + 1) % healthy.length;
    return healthy[this.rrIndex];
  }

  private pickLeastConnections(): PoolEntry {
    const healthy = this.entries.filter((e) => e.healthy);
    if (!healthy.length) throw new Error("All pool instances are unhealthy");
    return healthy.reduce((a, b) => (a.inFlight <= b.inFlight ? a : b));
  }

  private pick(): PoolEntry {
    return this.strategy === "round_robin"
      ? this.pickRoundRobin()
      : this.pickLeastConnections();
  }

  async callTool(
    toolName: string,
    toolInput: Record<string, unknown>
  ): Promise<string> {
    const entry = this.pick();
    entry.inFlight++;
    try {
      const result = await entry.client.callTool({ name: toolName, arguments: toolInput });
      const content = result.content as Array<{ type: string; text?: string }>;
      return content.find((c) => c.type === "text")?.text ?? "";
    } catch (e) {
      entry.healthy = false;
      throw e;
    } finally {
      entry.inFlight = Math.max(0, entry.inFlight - 1);
    }
  }

  async close(): Promise<void> {
    clearInterval(this.healthInterval);
    for (const { client, transport } of this.entries) {
      await client.close();
      await transport.close();
    }
  }
}

// ── Usage ─────────────────────────────────────────────────────────────────
const pool = new LoadBalancedMCPPool("least_connections");
await pool.addInstance("node", ["compute_server.js"], "compute-0");
await pool.addInstance("node", ["compute_server.js"], "compute-1");
await pool.addInstance("node", ["compute_server.js"], "compute-2");
pool.startHealthChecks();

const [r1, r2] = await Promise.all([
  pool.callTool("fuzzy_match_score", { a: "Acme Inc", b: "Acme Corp" }),
  pool.callTool("calculate_risk_score", { entity_id: "E001" }),
]);
console.log(r1, r2);
await pool.close();
What Just Happened?
  • Three compute server instances were registered in the pool.
  • A background loop pings each instance every 30 s and marks it healthy or unhealthy.
  • When call_tool fires, the pool picks the instance with the fewest in-flight calls (least-connections strategy) and increments that counter atomically.
  • If the call throws, the entry is immediately marked unhealthy and the next call will skip it until the health check restores it.
💡 Note — FailoverFailover — automatically routing traffic away from a failed server instance to a healthy one, with no manual intervention required. vs. Retry

Failover (skip to next healthy instance) and retry (call the same instance again) are distinct. The pool above implements failover. If your tool calls are idempotent, add retry logic on top with exponential backoff. If they are not idempotent (e.g., send_email), use failover only.

With routing and load balancing in place, you're ready to see all the pieces working together in a complete three-server system — real servers, real tools, real agent loop.

6. Full Stack: Three-Server Agent

This section builds a complete orchestrated system used in the Public Records / UCC Data Engineering domain (Capstone C3). Three independent servers, one host, one agent loop. All code is copy-paste runnable.

DataServer — data_server.py

WHAT
Exposes two tools: search_filings (text search over a UCC filing store) and get_filing_details (fetch a single filing by ID). Uses an in-memory dict for demo; swap for a real DB in production.
Python — data_server.py
"""data_server.py — UCC filing search and retrieval."""
import json
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("DataServer")

# Demo data — replace with real DB connection
FILINGS = {
    "F001": {"id": "F001", "debtor": "Acme Corp", "secured_party": "First National Bank",
             "amount": 500_000, "state": "CA", "status": "active"},
    "F002": {"id": "F002", "debtor": "Acme Inc",  "secured_party": "Regional Credit Union",
             "amount": 125_000, "state": "CA", "status": "active"},
    "F003": {"id": "F003", "debtor": "GlobalTech LLC", "secured_party": "Capital Partners",
             "amount": 2_000_000, "state": "NY", "status": "terminated"},
}


@mcp.tool()
def search_filings(query: str, state: str = "", max_results: int = 10) -> str:
    """
    Search UCC filings by debtor name (case-insensitive substring match).
    Returns a JSON list of matching filings with id, debtor, amount, and status.
    """
    q = query.lower()
    results = [
        {"id": f["id"], "debtor": f["debtor"], "amount": f["amount"], "status": f["status"]}
        for f in FILINGS.values()
        if q in f["debtor"].lower() and (not state or f["state"] == state)
    ][:max_results]
    if not results:
        return json.dumps({"results": [], "message": f"No filings found for '{query}'"})
    return json.dumps({"results": results, "total": len(results)})


@mcp.tool()
def get_filing_details(filing_id: str) -> str:
    """Retrieve full details for a single UCC filing by its ID."""
    filing = FILINGS.get(filing_id)
    if not filing:
        return json.dumps({"error": f"Filing '{filing_id}' not found"})
    return json.dumps(filing)


if __name__ == "__main__":
    mcp.run()
TypeScript — data_server.ts
/** data_server.ts — UCC filing search and retrieval. */
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";

const server = new McpServer({ name: "DataServer", version: "1.0.0" });

const FILINGS: Record<string, { id: string; debtor: string; secured_party: string; amount: number; state: string; status: string }> = {
  F001: { id: "F001", debtor: "Acme Corp", secured_party: "First National Bank", amount: 500_000, state: "CA", status: "active" },
  F002: { id: "F002", debtor: "Acme Inc",  secured_party: "Regional Credit Union", amount: 125_000, state: "CA", status: "active" },
  F003: { id: "F003", debtor: "GlobalTech LLC", secured_party: "Capital Partners", amount: 2_000_000, state: "NY", status: "terminated" },
};

server.tool(
  "search_filings",
  "Search UCC filings by debtor name.",
  { query: z.string(), state: z.string().optional(), max_results: z.number().optional() },
  async ({ query, state = "", max_results = 10 }) => {
    const q = query.toLowerCase();
    const results = Object.values(FILINGS)
      .filter((f) => f.debtor.toLowerCase().includes(q) && (!state || f.state === state))
      .slice(0, max_results)
      .map(({ id, debtor, amount, status }) => ({ id, debtor, amount, status }));
    return {
      content: [{ type: "text" as const, text: JSON.stringify({ results, total: results.length }) }],
    };
  }
);

server.tool(
  "get_filing_details",
  "Retrieve full details for a single UCC filing by ID.",
  { filing_id: z.string() },
  async ({ filing_id }) => {
    const filing = FILINGS[filing_id];
    const text = filing ? JSON.stringify(filing) : JSON.stringify({ error: `Filing '${filing_id}' not found` });
    return { content: [{ type: "text" as const, text }] };
  }
);

await server.connect(new StdioServerTransport());

ComputeServer — compute_server.py

Python — compute_server.py
"""compute_server.py — fuzzy matching and risk scoring."""
import json
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("ComputeServer")


def _levenshtein(a: str, b: str) -> int:
    """Classic edit distance — no external deps."""
    if len(a) < len(b):
        a, b = b, a
    if not b:
        return len(a)
    prev = list(range(len(b) + 1))
    for i, ca in enumerate(a, 1):
        curr = [i]
        for j, cb in enumerate(b, 1):
            curr.append(min(prev[j] + 1, curr[j - 1] + 1, prev[j - 1] + (ca != cb)))
        prev = curr
    return prev[-1]


@mcp.tool()
def fuzzy_match_score(entity_a: str, entity_b: str) -> str:
    """
    Compute a 0-100 similarity score between two entity name strings.
    Uses normalized Levenshtein distance. Score 100 = identical, 0 = completely different.
    """
    if not entity_a or not entity_b:
        return json.dumps({"error": "Both entity_a and entity_b are required", "score": 0})
    a, b = entity_a.lower().strip(), entity_b.lower().strip()
    max_len = max(len(a), len(b))
    if max_len == 0:
        return json.dumps({"score": 100, "entity_a": entity_a, "entity_b": entity_b})
    dist = _levenshtein(a, b)
    score = round((1 - dist / max_len) * 100, 1)
    return json.dumps({"score": score, "entity_a": entity_a, "entity_b": entity_b, "edit_distance": dist})


@mcp.tool()
def calculate_risk_score(
    amount: float,
    status: str,
    match_score: float = 100.0,
) -> str:
    """
    Calculate a 0-100 lien risk score.
    Factors: filing amount, status (active=high risk, terminated=low), entity match confidence.
    """
    if amount < 0:
        return json.dumps({"error": "amount must be non-negative", "risk_score": None})
    # Normalize amount: 0 = $0, 100 = $5M+
    amount_score = min(amount / 50_000, 100.0)
    status_score = 80.0 if status == "active" else 10.0
    # Lower match_score increases uncertainty, reducing risk confidence
    confidence = match_score / 100.0
    raw = (amount_score * 0.5 + status_score * 0.5) * confidence
    risk = round(min(raw, 100.0), 1)
    return json.dumps({
        "risk_score": risk,
        "factors": {"amount_score": amount_score, "status_score": status_score, "confidence": confidence},
    })


if __name__ == "__main__":
    mcp.run()
TypeScript — compute_server.ts
/** compute_server.ts — fuzzy matching and risk scoring. */
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";

function levenshtein(a: string, b: string): number {
  if (a.length < b.length) [a, b] = [b, a];
  if (!b.length) return a.length;
  let prev = Array.from({ length: b.length + 1 }, (_, i) => i);
  for (let i = 1; i <= a.length; i++) {
    const curr = [i];
    for (let j = 1; j <= b.length; j++) {
      curr.push(Math.min(prev[j] + 1, curr[j - 1] + 1, prev[j - 1] + (a[i - 1] !== b[j - 1] ? 1 : 0)));
    }
    prev = curr;
  }
  return prev[prev.length - 1];
}

const server = new McpServer({ name: "ComputeServer", version: "1.0.0" });

server.tool(
  "fuzzy_match_score",
  "0-100 similarity score between two entity names.",
  { entity_a: z.string(), entity_b: z.string() },
  async ({ entity_a, entity_b }) => {
    const a = entity_a.toLowerCase().trim();
    const b = entity_b.toLowerCase().trim();
    const maxLen = Math.max(a.length, b.length);
    if (!maxLen) return { content: [{ type: "text" as const, text: JSON.stringify({ score: 100 }) }] };
    const dist = levenshtein(a, b);
    const score = Math.round((1 - dist / maxLen) * 1000) / 10;
    return { content: [{ type: "text" as const, text: JSON.stringify({ score, entity_a, entity_b, edit_distance: dist }) }] };
  }
);

server.tool(
  "calculate_risk_score",
  "Calculate a 0-100 lien risk score.",
  { amount: z.number(), status: z.string(), match_score: z.number().optional() },
  async ({ amount, status, match_score = 100 }) => {
    const amountScore = Math.min(amount / 50_000, 100);
    const statusScore = status === "active" ? 80 : 10;
    const confidence = match_score / 100;
    const risk = Math.min(Math.round((amountScore * 0.5 + statusScore * 0.5) * confidence * 10) / 10, 100);
    return { content: [{ type: "text" as const, text: JSON.stringify({ risk_score: risk, factors: { amountScore, statusScore, confidence } }) }] };
  }
);

await server.connect(new StdioServerTransport());

StorageServer — storage_server.py

Python — storage_server.py
"""
storage_server.py — save and recall analysis results.
In-memory for demo. Replace save/recall with ChromaDB for production vector search.
"""
import json
import time
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("StorageServer")

# In-memory store: {result_id: {entity, risk_score, timestamp, ...}}
RESULTS: dict[str, dict] = {}


@mcp.tool()
def save_result(
    entity_name: str,
    risk_score: float,
    filing_ids: list[str],
    notes: str = "",
) -> str:
    """
    Persist an entity risk analysis result.
    Returns the result_id that can be used for later retrieval.
    """
    if not entity_name:
        return json.dumps({"error": "entity_name is required"})
    result_id = f"R{len(RESULTS) + 1:04d}"
    RESULTS[result_id] = {
        "result_id": result_id,
        "entity_name": entity_name,
        "risk_score": risk_score,
        "filing_ids": filing_ids,
        "notes": notes,
        "created_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    }
    return json.dumps({"result_id": result_id, "saved": True})


@mcp.tool()
def recall_similar(entity_name: str, min_risk_score: float = 0.0) -> str:
    """
    Retrieve previously saved results for entities similar to entity_name.
    Filters by optional minimum risk score threshold.
    """
    q = entity_name.lower()
    matches = [
        r for r in RESULTS.values()
        if q in r["entity_name"].lower() and r["risk_score"] >= min_risk_score
    ]
    return json.dumps({"matches": matches, "total": len(matches)})


if __name__ == "__main__":
    mcp.run()
TypeScript — storage_server.ts
/** storage_server.ts — save and recall analysis results. */
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";

const RESULTS = new Map<string, { result_id: string; entity_name: string; risk_score: number; filing_ids: string[]; notes: string; created_at: string }>();

const server = new McpServer({ name: "StorageServer", version: "1.0.0" });

server.tool(
  "save_result",
  "Persist an entity risk analysis result.",
  { entity_name: z.string(), risk_score: z.number(), filing_ids: z.array(z.string()), notes: z.string().optional() },
  async ({ entity_name, risk_score, filing_ids, notes = "" }) => {
    const result_id = `R${String(RESULTS.size + 1).padStart(4, "0")}`;
    RESULTS.set(result_id, { result_id, entity_name, risk_score, filing_ids, notes, created_at: new Date().toISOString() });
    return { content: [{ type: "text" as const, text: JSON.stringify({ result_id, saved: true }) }] };
  }
);

server.tool(
  "recall_similar",
  "Retrieve saved results for entities similar to entity_name.",
  { entity_name: z.string(), min_risk_score: z.number().optional() },
  async ({ entity_name, min_risk_score = 0 }) => {
    const q = entity_name.toLowerCase();
    const matches = [...RESULTS.values()].filter((r) => r.entity_name.toLowerCase().includes(q) && r.risk_score >= min_risk_score);
    return { content: [{ type: "text" as const, text: JSON.stringify({ matches, total: matches.length }) }] };
  }
);

await server.connect(new StdioServerTransport());

Putting It All Together — orchestrated_agent.py

WHAT
The orchestrated agent connects all three servers via MCPHost, configures SmartRouter with intent routing, and runs an entity resolution query. Claude transparently calls tools from all three servers in one conversation.
Python — orchestrated_agent.py
"""
orchestrated_agent.py — three-server agent for UCC entity resolution.
WHAT: Connects DataServer + ComputeServer + StorageServer via MCPHost.
      Uses SmartRouter with intent routing to minimize tool list per turn.
WHY:  The LLM sees only the tools it needs; results are persisted automatically.
"""
import asyncio

# Import MCPHost and SmartRouter from Sections 3 and 4
from mcp_host import MCPHost
from smart_router import SmartRouter, RoutingStrategy


async def main():
    host = MCPHost()

    # Connect all three servers
    await host.add_server("db",    "python", ["data_server.py"])
    await host.add_server("ml",    "python", ["compute_server.py"])
    await host.add_server("store", "python", ["storage_server.py"])

    # Configure intent router so the classifier knows which tools live where
    router = SmartRouter(host, strategy=RoutingStrategy.INTENT)
    router.configure_server_tools({
        "db":    ["search_filings", "get_filing_details"],
        "ml":    ["fuzzy_match_score", "calculate_risk_score"],
        "store": ["save_result", "recall_similar"],
    })

    system = """You are an expert UCC lien risk analyst.
When given an entity name:
1. Search for filings using db::search_filings
2. For each match, score similarity with ml::fuzzy_match_score
3. Calculate risk with ml::calculate_risk_score
4. Save all high-risk results (score > 60) with store::save_result
5. Check for prior analyses with store::recall_similar
Provide a structured risk summary at the end."""

    answer = await router.run_agent(
        user_message="Analyze lien risk for 'Acme Corporation' — are there any high-risk filings?",
        system_prompt=system,
        max_turns=15,
    )
    print("=== Agent Answer ===")
    print(answer)

    await host.close()


asyncio.run(main())
TypeScript — orchestrated_agent.ts
/** orchestrated_agent.ts — three-server agent for UCC entity resolution. */
import { MCPHost } from "./mcp-host.js";
import { SmartRouter } from "./smart-router.js";

const host = new MCPHost();
await host.addServer("db",    "node", ["data_server.js"]);
await host.addServer("ml",    "node", ["compute_server.js"]);
await host.addServer("store", "node", ["storage_server.js"]);

const router = new SmartRouter(host, "intent");
router.configureServerTools({
  db:    ["search_filings", "get_filing_details"],
  ml:    ["fuzzy_match_score", "calculate_risk_score"],
  store: ["save_result", "recall_similar"],
});

const system = `You are an expert UCC lien risk analyst.
When given an entity name:
1. Search for filings using db::search_filings
2. Score entity similarity with ml::fuzzy_match_score
3. Calculate risk with ml::calculate_risk_score
4. Save high-risk results (score > 60) with store::save_result
5. Check for prior analyses with store::recall_similar
Provide a structured risk summary.`;

const answer = await router.runAgent(
  "Analyze lien risk for 'Acme Corporation' — any high-risk filings?",
  system,
  15
);
console.log("=== Agent Answer ===");
console.log(answer);
await host.close();
What Just Happened?
  • The intent classifier determined the query needed db and ml (and possibly store), so those tools were shown first.
  • Claude called db::search_filings → got F001, F002; called ml::fuzzy_match_score for each → high similarity on both.
  • Called ml::calculate_risk_score for each filing; F001 scored 73 (high risk).
  • Called store::save_result for F001; called store::recall_similar to check history.
  • Returned a structured summary — all within a single run_agent call, spanning three physical server processes.

7. Observability for Multi-Server Systems

Distributed tracingDistributed tracing — a technique where a unique trace_id is generated at the start of an agent run and propagated through every tool call across every server. Each server logs the trace_id alongside its own operation logs, so you can reconstruct the full call graph after the fact. is the difference between "something is slow" and "the StorageServer's recall_similar takes 800 ms on entity names with spaces." Without cross-server trace IDs, debugging multi-server agents is guesswork.

WHAT
TracedMCPHost subclasses MCPHost and injects a trace_id into every tool call. Each server logs the trace_id on receipt. The host collects per-server latency and error data. At the end of a run, get_trace_summary() shows a full call graph.
WHY
In a real agent run that calls 8-12 tools across 3 servers, you need to know: which server was the bottleneck, which tool call failed, and how latency is distributed. Without trace_id propagation this data is unreachable — each server's logs have no cross-server join key.
Python — TracedMCPHost
"""
TracedMCPHost — distributed tracing for multi-server MCP agents.
WHAT: Injects trace_id into every tool call; collects per-server latency.
WHY:  Identify bottlenecks, reconstruct call graphs, isolate errors by server.
GOTCHA: trace_id must be a string, not a UUID object, for JSON serialization.
"""
import asyncio
import time
import uuid
from dataclasses import dataclass, field
from typing import Optional

# TracedMCPHost extends MCPHost from Section 3
from mcp_host import MCPHost


@dataclass
class SpanRecord:
    trace_id: str
    span_id: str
    server_alias: str
    tool_name: str
    start_time: float
    end_time: Optional[float] = None
    success: bool = True
    error: Optional[str] = None

    @property
    def duration_ms(self) -> float:
        if self.end_time is None:
            return 0.0
        return (self.end_time - self.start_time) * 1000


class TracedMCPHost(MCPHost):
    """
    Subclass of MCPHost with distributed tracing.
    Each agent run gets a unique trace_id; each tool call gets a span_id.
    """

    def __init__(self):
        super().__init__()
        self._spans: list[SpanRecord] = []
        self._current_trace_id: Optional[str] = None

    def start_trace(self, trace_id: Optional[str] = None) -> str:
        """Begin a new agent run trace. Returns the trace_id."""
        self._current_trace_id = trace_id or str(uuid.uuid4())[:8]
        self._spans.clear()
        print(f"[Trace] Starting run trace_id={self._current_trace_id}")
        return self._current_trace_id

    async def call_tool(self, namespaced_name: str, tool_input: dict) -> str:
        """
        Override call_tool to inject trace metadata and record latency.
        WHY: Every tool call is instrumented without changing individual servers.
        """
        if "::" not in namespaced_name:
            raise ValueError(f"Tool name '{namespaced_name}' missing namespace prefix")
        alias = namespaced_name.split("::")[0]
        span_id = str(uuid.uuid4())[:8]
        span = SpanRecord(
            trace_id=self._current_trace_id or "untraced",
            span_id=span_id,
            server_alias=alias,
            tool_name=namespaced_name,
            start_time=time.time(),
        )
        self._spans.append(span)

        # Inject trace metadata — servers that support it can log it
        traced_input = {**tool_input, "_trace_id": span.trace_id, "_span_id": span_id}

        try:
            result = await super().call_tool(namespaced_name, traced_input)
            span.end_time = time.time()
            print(f"  [Span] {namespaced_name} → {span.duration_ms:.1f}ms OK  trace={span.trace_id}")
            return result
        except Exception as e:
            span.end_time = time.time()
            span.success = False
            span.error = str(e)
            print(f"  [Span] {namespaced_name} → {span.duration_ms:.1f}ms ERR {e}  trace={span.trace_id}")
            raise

    def get_trace_summary(self) -> dict:
        """
        Return a full trace summary with per-server latency breakdown.
        WHY: One dict to paste into your log aggregator (Datadog, CloudWatch, etc.)
        """
        if not self._spans:
            return {"trace_id": self._current_trace_id, "spans": [], "total_ms": 0}

        total_ms = sum(s.duration_ms for s in self._spans)
        by_server: dict[str, dict] = {}
        for span in self._spans:
            if span.server_alias not in by_server:
                by_server[span.server_alias] = {"calls": 0, "total_ms": 0.0, "errors": 0}
            by_server[span.server_alias]["calls"] += 1
            by_server[span.server_alias]["total_ms"] += span.duration_ms
            if not span.success:
                by_server[span.server_alias]["errors"] += 1

        return {
            "trace_id": self._current_trace_id,
            "total_calls": len(self._spans),
            "total_ms": round(total_ms, 1),
            "by_server": {
                alias: {
                    "calls": data["calls"],
                    "avg_ms": round(data["total_ms"] / data["calls"], 1),
                    "errors": data["errors"],
                }
                for alias, data in by_server.items()
            },
            "spans": [
                {
                    "span_id": s.span_id,
                    "tool": s.tool_name,
                    "duration_ms": round(s.duration_ms, 1),
                    "success": s.success,
                    "error": s.error,
                }
                for s in self._spans
            ],
        }


# ── Usage ─────────────────────────────────────────────────────────────────
async def main():
    host = TracedMCPHost()
    await host.add_server("db",    "python", ["data_server.py"])
    await host.add_server("ml",    "python", ["compute_server.py"])
    await host.add_server("store", "python", ["storage_server.py"])

    trace_id = host.start_trace()
    print(f"Agent run: trace_id={trace_id}")

    # Run some tool calls (or use SmartRouter.run_agent with host=traced_host)
    await host.call_tool("db::search_filings", {"query": "Acme"})
    await host.call_tool("ml::fuzzy_match_score", {"entity_a": "Acme Corp", "entity_b": "Acme Inc"})
    await host.call_tool("store::save_result", {"entity_name": "Acme Corp", "risk_score": 73.0, "filing_ids": ["F001"]})

    import json
    summary = host.get_trace_summary()
    print("\n=== Trace Summary ===")
    print(json.dumps(summary, indent=2))

    await host.close()


asyncio.run(main())
TypeScript — TracedMCPHost
/**
 * TracedMCPHost — distributed tracing for multi-server MCP agents.
 * WHAT: Injects trace_id into every call; collects per-server latency.
 */
import { MCPHost } from "./mcp-host.js";
import { randomBytes } from "crypto";

interface SpanRecord {
  traceId: string;
  spanId: string;
  serverAlias: string;
  toolName: string;
  startTime: number;
  endTime?: number;
  success: boolean;
  error?: string;
}

export class TracedMCPHost extends MCPHost {
  private spans: SpanRecord[] = [];
  private currentTraceId?: string;

  startTrace(traceId?: string): string {
    this.currentTraceId = traceId ?? randomBytes(4).toString("hex");
    this.spans = [];
    console.log(`[Trace] Starting run trace_id=${this.currentTraceId}`);
    return this.currentTraceId;
  }

  async callTool(
    namespacedName: string,
    toolInput: Record<string, unknown>
  ): Promise<string> {
    const alias = namespacedName.split("::")[0];
    const spanId = randomBytes(4).toString("hex");
    const span: SpanRecord = {
      traceId: this.currentTraceId ?? "untraced",
      spanId,
      serverAlias: alias,
      toolName: namespacedName,
      startTime: Date.now(),
      success: true,
    };
    this.spans.push(span);

    // Inject trace metadata for servers that log it
    const tracedInput = { ...toolInput, _trace_id: span.traceId, _span_id: spanId };
    try {
      const result = await super.callTool(namespacedName, tracedInput);
      span.endTime = Date.now();
      console.log(`  [Span] ${namespacedName} → ${span.endTime - span.startTime}ms OK`);
      return result;
    } catch (e) {
      span.endTime = Date.now();
      span.success = false;
      span.error = String(e);
      console.error(`  [Span] ${namespacedName} → ${span.endTime! - span.startTime}ms ERR ${e}`);
      throw e;
    }
  }

  getTraceSummary() {
    const totalMs = this.spans.reduce(
      (acc, s) => acc + ((s.endTime ?? s.startTime) - s.startTime), 0
    );
    const byServer: Record<string, { calls: number; totalMs: number; errors: number }> = {};
    for (const span of this.spans) {
      byServer[span.serverAlias] ??= { calls: 0, totalMs: 0, errors: 0 };
      byServer[span.serverAlias].calls++;
      byServer[span.serverAlias].totalMs += (span.endTime ?? span.startTime) - span.startTime;
      if (!span.success) byServer[span.serverAlias].errors++;
    }
    return {
      trace_id: this.currentTraceId,
      total_calls: this.spans.length,
      total_ms: totalMs,
      by_server: Object.fromEntries(
        Object.entries(byServer).map(([alias, d]) => [
          alias,
          { calls: d.calls, avg_ms: Math.round(d.totalMs / d.calls), errors: d.errors },
        ])
      ),
    };
  }
}

// ── Usage ─────────────────────────────────────────────────────────────────
const host = new TracedMCPHost();
await host.addServer("db",    "node", ["data_server.js"]);
await host.addServer("ml",    "node", ["compute_server.js"]);
await host.addServer("store", "node", ["storage_server.js"]);

const traceId = host.startTrace();
await host.callTool("db::search_filings", { query: "Acme" });
await host.callTool("ml::fuzzy_match_score", { entity_a: "Acme Corp", entity_b: "Acme Inc" });

console.log(JSON.stringify(host.getTraceSummary(), null, 2));
await host.close();
📈 Why It Matters — Real Numbers

A production UCC entity resolution agent at a legal data firm averaged 11 tool calls per agent run across 3 servers. Before tracing, debugging an intermittent 4-second slowdown took 2 days. After adding TracedMCPHost, they identified that store::recall_similar was performing a full table scan on entity names with ampersands (&) due to a missing index — 15-minute fix once identified. Distributed tracing converted a guessing game into a 30-second diagnosis.

8. Lab: Build a Three-Server Orchestrated Agent

Follow these steps in order. Each step has a command, expected output, and a checkpoint. If you see the expected output, proceed. If not, check the troubleshooting note.

STEP 1

Start the DataServer

Save data_server.py from Section 6 and verify it starts cleanly.

Terminal 1 — DataServer
python data_server.py
Starting MCP server 'DataServer' with transport 'stdio'

Leave this terminal open. The server runs until you close it.

STEP 2

Start the ComputeServer

In a second terminal window, start the compute server.

Terminal 2 — ComputeServer
python compute_server.py
Starting MCP server 'ComputeServer' with transport 'stdio'
STEP 3

Start the StorageServer

Terminal 3 — StorageServer
python storage_server.py
Starting MCP server 'StorageServer' with transport 'stdio'
STEP 4

Create MCPHost and connect all three

Save the following as orchestrated_agent.py (the full version from Section 6). Then install dependencies.

Terminal 4 — Host
pip install mcp anthropic
Successfully installed anthropic-0.x.x mcp-1.x.x
STEP 5

Run an entity resolution query

Verify that tool calls span all three servers in one agent run.

Terminal 4 — Agent
python orchestrated_agent.py
=== Agent Answer === Based on my analysis of UCC filings for "Acme Corporation": **Filings Found (2 matches):** - F001: Acme Corp — $500,000 active lien (similarity: 91%) → Risk Score: 73.2 (HIGH) - F002: Acme Inc — $125,000 active lien (similarity: 78%) → Risk Score: 41.5 (MEDIUM) **High-Risk Result Saved:** R0001 (Acme Corp, score 73.2) **Prior Analyses:** None found for this entity. **Recommendation:** F001 presents a high-risk active lien. Recommend manual review before extending credit.
STEP 6

Verify server-level routing in logs

Add TracedMCPHost from Section 7 (replace MCPHost with TracedMCPHost in your agent) and re-run. Verify the trace summary shows calls distributed across all three servers.

Expected trace summary
=== Trace Summary === { "trace_id": "a3f1b2c4", "total_calls": 6, "total_ms": 312.4, "by_server": { "db": { "calls": 2, "avg_ms": 18, "errors": 0 }, "ml": { "calls": 2, "avg_ms": 12, "errors": 0 }, "store": { "calls": 2, "avg_ms": 9, "errors": 0 } } }

Checkpoint: If you see calls distributed across all three servers in the trace summary, your orchestrated multi-server agent is working correctly.

⚠ Troubleshooting
  • Server not found: Make sure each *_server.py file is in the same directory as orchestrated_agent.py.
  • Tool namespace error: Verify MCPHost.add_server() aliases match your configure_server_tools() keys ("db", "ml", "store").
  • ANTHROPIC_API_KEY missing: Run export ANTHROPIC_API_KEY=sk-ant-... before starting the agent.
  • Agent uses wrong server: Check that your intent router alias list matches the actual tool lists. A mismatch causes the classifier to exclude a server.

9. Knowledge Check

Five questions. Each reveals the answer immediately with an explanation.

1. You have 40 tools across 4 servers and your agent frequently calls the wrong server. Which routing strategy addresses this most directly?

AStatic map — pre-define every tool→server pair
BRound-robin load balancing across all servers
CIntent routing — classify the query first, show only relevant tools
DCapability matching with a sorted tool list
The static map solves routing but doesn't reduce the tool list. The model still sees all 40 tools and the problem is tool list noise, not incorrect routing logic.
Intent routing classifies the query and shows only the tools of relevant servers (e.g., 10 of 40). This directly reduces the noise that causes the model to pick the wrong server.

2. Two servers both expose a tool named search. Without namespacing, what happens when the LLM calls search?

AThe host calls both servers simultaneously and merges results
BThe first server registered wins — silently
CThe host raises an error and aborts the agent run
DThe LLM is prompted to disambiguate before the call is made
The host does not merge results or ask the LLM to choose. There is no automatic ambiguity resolution in the base MCP protocol.
Without namespacing, the host's internal tool map uses tool name as key and simply overwrites the first entry with the second. This is a silent bug — the wrong server gets called with no error.

3. Which load balancing strategy should you use when your tool calls have highly variable latency (some take 10 ms, others 2 seconds)?

ARound-robin, because it's stateless and predictable
BLeast-connections, because slow calls won't saturate one instance
CRandom selection to avoid any pattern
DStatic routing, because latency doesn't affect the correct server choice
Round-robin doesn't account for in-flight call count. If one instance has 5 slow calls running, round-robin will still send new calls to it at the same rate as an idle instance.
Least-connections tracks in-flight call count per instance and prefers the least-loaded one. When calls vary from 10 ms to 2 s, this prevents a single instance from becoming a bottleneck.

4. What is the key advantage of the hierarchical topology over fan-out?

AIt is faster because there are fewer server processes to start
BIt allows pipeline-style sequential processing
CThe supervisor server exposes abstract tools, hiding sub-server complexity from the LLM
DIt automatically handles tool name collisions without explicit namespacing
Hierarchical topology does not reduce process count or enable sequential processing. It actually adds a network hop compared to fan-out, making it slightly slower.
The supervisor server presents a simplified API (e.g., analyze_entity) to the LLM while internally calling multiple sub-servers. This reduces tool list size and hides internal orchestration complexity.

5. In TracedMCPHost, why is in_flight decremented in a finally block rather than in the success path only?

APython requires finally for any variable decrement
BIf the call throws an exception, the counter must still be decremented or it leaks permanently
Cfinally runs faster than the success path for numeric operations
DTo prevent the health check loop from reading a stale counter value
Python has no such requirement — finally is a design choice here, not a language constraint.
If the tool call throws, Python will skip any code after the raise in the try block. Without finally, the counter stays permanently incremented — the least-connections strategy will think the instance is busy forever and route around it, effectively removing it from the pool.

6. (Bonus) Your intent classifier returns ["db"] for a query, but the correct answer also needs the ml server. The agent fails. What is the safest default fix?

AAlways show all tools, ignoring the classifier
BFall back to the full tool list when the classifier returns fewer than N servers
CRun the classifier three times and take a majority vote
DAdd a retry loop that re-classifies if the agent hits an error
Always showing all tools defeats the purpose of intent routing and re-introduces the tool list noise problem.
A conservative threshold (e.g., "if classifier returns fewer than 2 servers, show all tools") preserves the benefits of intent routing for common cases while gracefully degrading for ambiguous queries. This is safer than retrying, which adds latency and may loop.