Multi-Server Orchestration
Connect multiple specialized MCP servers into a single coherent agent. Learn topology patterns, tool namespacing, smart routing, load balancing, and distributed observability — all with complete Python and TypeScript implementations.
- ✓Explain the three topology patterns (fan-out, pipeline, hierarchical) and when to use each
- ✓Implement prefix-based and schema-based tool namespacing to prevent collisions
- ✓Build a SmartRouter with static map, capability match, and intent routing strategies
- ✓Implement load balancing with round-robin, least-connections, and automatic failover
- ✓Add distributed tracing to reconstruct cross-server call graphs
1. Why Orchestrate Multiple Servers?
BEFORE: Imagine you hire one expert consultant for a complex project. They are world-class at financial modeling — but when you need a marketing strategy or a legal opinion, they shrug. You hired a specialist, not a generalist.
PAIN: A single MCP server is exactly that specialist. It exposes the tools of one domain perfectly — your database server handles every SQL query with precision — but the moment your agent needs to score a risk metric, store a result, or fire a Slack notification, that server has nothing to offer. The LLM sees a narrow toolset and is forced to either hallucinate or give up.
MAPPING: Orchestrating multiple MCP servers gives your agent a full consulting team: one specialist for data, one for compute, one for storage, one for communications. The orchestrating host surfaces all their tools simultaneously, and the LLM picks the right expert for each sub-task — just as a project manager routes work to the right consultant without asking the client to manage the logistics.
In practice, a production agent almost always needs at least four capability domains:
| Domain | Typical Server | Example Tools |
|---|---|---|
| Data | DataServer | search_filings, get_filing_details, run_sql |
| Compute | ComputeServer | fuzzy_match_score, calculate_risk, ml_classify |
| Storage | StorageServer | save_result, recall_similar, delete_record |
| Communications | CommsServer | send_email, post_slack, create_ticket |
Teams at Anthropic building production agents report that a typical enterprise workflow touches 3–5 MCP servers per agent run. Without orchestration, each domain becomes a separate agent with its own context window — multiplying latency and cost. A single well-orchestrated host cuts round-trips by 60-80% compared to chaining separate agent calls.
MCP orchestration means a single MCP hostMCP Host — the process that manages connections to one or more MCP servers, aggregates their tool lists, and routes tool calls. Claude Desktop and Claude Code are hosts; you can also build a custom host in Python or TypeScript. maintains simultaneous connections to N servers, merges their tool schemas into one list that the LLM sees, and dispatches each tool call to the correct server — all transparently to the model.
2. Topology Patterns
There are three canonical ways to wire multiple servers together. Choosing the wrong topology for your workload is one of the most common architectural mistakes — each has clear trade-offs you should understand before writing a line of code.
Fan-out: All Tools Visible Simultaneously
The host connects to all N servers on startup and merges their tool lists. The LLM sees every tool in a single list and picks freely. Best for: general-purpose agents where you cannot predict which capabilities will be needed per turn.
- Trade-off: Large tool lists (30+ tools) slightly increase prompt token count and can confuse smaller models. Mitigate with intent routing (Section 4).
Pipeline: Chained Processing
Server A's output becomes Server B's input. The host enforces a processing order — it calls Server A, passes the structured result to Server B, and returns the final output. Best for: ETL-style workflows where step N always feeds step N+1.
- Trade-off: Inflexible. If the agent decides mid-run that it needs to skip a stage, the pipeline fights you. Reserve for truly sequential transformations.
Hierarchical: Supervisor + Sub-Servers
A supervisor serverSupervisor Server — an MCP server that is itself an MCP client. It exposes high-level abstract tools to the LLM (e.g., analyze_entity), but internally calls sub-servers to fulfill them. The LLM never sees the sub-server complexity. acts as a client to a set of sub-servers. It exposes only high-level abstract tools to the host LLM. Best for: capability abstraction, where you want to present a simplified API to the model and hide internal orchestration logic.
- Trade-off: Adds a network hop. The supervisor must be a full MCP server+client simultaneously — more code to maintain. Worth it when tool list noise is a real problem.
| Pattern | Tool Visibility | Best For | Main Risk |
|---|---|---|---|
| Fan-out | All tools merged | General agents | Tool list bloat |
| Pipeline | Sequential exposure | ETL / transforms | Inflexibility |
| Hierarchical | Abstract facade | Capability abstraction | Extra hop latency |
search? Tool namespacing solves this before it ever reaches the router.3. Tool Namespacing
When two servers both expose search, the host has a collision. Without disambiguation, the first server registered wins — silently. Tool namespacingTool namespacing — a strategy for making tool names globally unique across multiple servers by prepending a server identifier. Prevents silent collision where the wrong server handles a tool call. makes every tool name globally unique so the router always calls the right server.
Strategy 1: Prefix-based Namespacing
The host rewrites tool names on ingestion: search from the DB server becomes db::search; search from the ML server becomes ml::search. The LLM sees prefixed names; the host strips the prefix before forwarding the call.
MCPHost.add_server() registers a server with an alias. _build_namespaced_tools() rewrites the tool list. call_tool() parses the :: separator and routes to the right client.
"""
Prefix-based tool namespacing: db::search, ml::search
Each server gets an alias; the host rewrites tool names on registration.
"""
import asyncio
import json
from contextlib import AsyncExitStack
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from mcp.types import Tool
class MCPHost:
"""
WHAT: Manages N server connections; rewrites tool names with server prefixes.
WHY: Prevents silent collisions when two servers expose the same tool name.
"""
def __init__(self):
self._sessions: dict[str, ClientSession] = {}
self._namespaced_tools: list[dict] = [] # merged list sent to LLM
self._tool_to_server: dict[str, str] = {} # namespaced_name → alias
self._exit_stack = AsyncExitStack()
async def add_server(
self,
alias: str, # e.g. "db", "ml", "storage"
command: str, # e.g. "python"
args: list[str], # e.g. ["data_server.py"]
env: dict | None = None,
) -> None:
"""
WHAT: Connect to a stdio MCP server and register its tools under alias::.
GOTCHA: alias must be unique; duplicate aliases overwrite the previous entry.
"""
params = StdioServerParameters(command=command, args=args, env=env)
read, write = await self._exit_stack.enter_async_context(
stdio_client(params)
)
session = await self._exit_stack.enter_async_context(
ClientSession(read, write)
)
await session.initialize()
self._sessions[alias] = session
# Fetch tools and namespace them
tools_result = await session.list_tools()
for tool in tools_result.tools:
namespaced_name = f"{alias}::{tool.name}"
self._namespaced_tools.append({
"name": namespaced_name,
"description": f"[{alias}] {tool.description}",
"input_schema": tool.inputSchema,
})
self._tool_to_server[namespaced_name] = alias
def get_tools_for_llm(self) -> list[dict]:
"""Return the full namespaced tool list to pass to the Anthropic API."""
return [
{
"name": t["name"],
"description": t["description"],
"input_schema": t["input_schema"],
}
for t in self._namespaced_tools
]
async def call_tool(self, namespaced_name: str, tool_input: dict) -> str:
"""
WHAT: Parse alias::tool_name, look up the right session, forward the call.
WHY: The LLM always uses the namespaced name; this is the single routing point.
GOTCHA: If alias is unknown, raise immediately — never silently drop calls.
"""
if "::" not in namespaced_name:
raise ValueError(
f"Tool name '{namespaced_name}' missing namespace prefix (expected alias::tool_name)"
)
alias, original_name = namespaced_name.split("::", 1)
if alias not in self._sessions:
raise KeyError(f"No server registered with alias '{alias}'")
session = self._sessions[alias]
result = await session.call_tool(original_name, tool_input)
# Extract text content from MCP result
if result.content:
return result.content[0].text if hasattr(result.content[0], "text") else str(result.content[0])
return ""
async def close(self) -> None:
await self._exit_stack.aclose()
# ── Usage ────────────────────────────────────────────────────────────────────
async def demo():
host = MCPHost()
try:
await host.add_server("db", "python", ["data_server.py"])
await host.add_server("ml", "python", ["compute_server.py"])
await host.add_server("store", "python", ["storage_server.py"])
tools = host.get_tools_for_llm()
# Tools are now: db::search_filings, ml::fuzzy_match_score, store::save_result, …
for t in tools:
print(f" {t['name']}: {t['description'][:60]}")
# Route a call — alias extracted automatically
result = await host.call_tool("db::search_filings", {"query": "Acme Corp"})
print("Result:", result)
finally:
await host.close()
asyncio.run(demo())
/**
* Prefix-based tool namespacing: db::search, ml::search
* WHAT: MCPHost manages N stdio servers and rewrites tool names.
* WHY: The LLM always emits alias::toolName, so routing is O(1).
*/
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";
import { Tool } from "@modelcontextprotocol/sdk/types.js";
interface ServerEntry {
client: Client;
transport: StdioClientTransport;
}
interface NamespacedTool {
name: string;
description: string;
input_schema: Record;
}
export class MCPHost {
private sessions = new Map<string, ServerEntry>();
private namespacedTools: NamespacedTool[] = [];
private toolToServer = new Map<string, string>(); // namespaced → alias
async addServer(
alias: string,
command: string,
args: string[],
env?: Record<string, string>
): Promise<void> {
const transport = new StdioClientTransport({ command, args, env });
const client = new Client({ name: "mcp-host", version: "1.0.0" }, {});
await client.connect(transport);
this.sessions.set(alias, { client, transport });
// Fetch and namespace tools
const { tools } = await client.listTools();
for (const tool of tools) {
const namespacedName = `${alias}::${tool.name}`;
this.namespacedTools.push({
name: namespacedName,
description: `[${alias}] ${tool.description ?? ""}`,
input_schema: (tool.inputSchema as Record<string, unknown>) ?? {},
});
this.toolToServer.set(namespacedName, alias);
}
}
getToolsForLLM(): NamespacedTool[] {
return [...this.namespacedTools];
}
async callTool(
namespacedName: string,
toolInput: Record<string, unknown>
): Promise<string> {
if (!namespacedName.includes("::")) {
throw new Error(
`Tool '${namespacedName}' missing namespace prefix (expected alias::toolName)`
);
}
const [alias, originalName] = namespacedName.split("::", 2);
const entry = this.sessions.get(alias);
if (!entry) throw new Error(`No server registered with alias '${alias}'`);
const result = await entry.client.callTool({
name: originalName,
arguments: toolInput,
});
const content = result.content as Array<{ type: string; text?: string }>;
return content.find((c) => c.type === "text")?.text ?? "";
}
async close(): Promise<void> {
for (const { client, transport } of this.sessions.values()) {
await client.close();
await transport.close();
}
}
}
// ── Usage ─────────────────────────────────────────────────────────────────
const host = new MCPHost();
await host.addServer("db", "node", ["data_server.js"]);
await host.addServer("ml", "node", ["compute_server.js"]);
await host.addServer("store", "node", ["storage_server.js"]);
const tools = host.getToolsForLLM();
// → db::search_filings, ml::fuzzy_match_score, store::save_result …
console.log(tools.map((t) => t.name));
const result = await host.callTool("db::search_filings", { query: "Acme Corp" });
console.log("Result:", result);
await host.close();
When add_server("db", ...) runs, it fetches the raw tool list from the DB server (search_filings, get_filing_details) and rewrites each name to db::search_filings, db::get_filing_details. When the LLM later emits db::search_filings, the host splits on ::, finds the db session, and calls search_filings on it — the server never sees the prefix.
Strategy 2: Schema-based Namespacing
Instead of rewriting names, include a server field in the tool's input_schema metadata. The router reads this field from the LLM's tool call to determine destination. Less common — use prefix-based unless you have a specific reason to keep original tool names.
Schema-based namespacing requires the LLM to correctly fill in the server field on every call. Models occasionally omit optional fields. Prefix-based puts the routing info in the name, which the model always provides.
4. Routing Strategies
The host must decide which server to call. The choice of routing strategy affects latency, reliability, and tool list size. Three strategies cover virtually every real use case.
SmartRouter wraps the MCPHost and selects a routing strategy at call time. Three strategies: static (dict lookup), capability (match tool name in per-server list), and intent (run a fast LLM classification first).
haiku for the classifier — it costs ~200x less than Sonnet and classification accuracy is equivalent.
"""
SmartRouter — three configurable routing strategies.
WHAT: Wraps MCPHost and selects server-specific tool subsets based on strategy.
WHY: Reduces tool list noise, improves routing accuracy, lowers token cost.
"""
import anthropic
from enum import Enum
from typing import Literal
class RoutingStrategy(str, Enum):
STATIC = "static" # fastest: pre-built name→server dict
CAPABILITY = "capability" # flexible: match tool name in per-server list
INTENT = "intent" # smartest: classify intent, show only relevant tools
class SmartRouter:
def __init__(
self,
host: "MCPHost", # from Section 3
strategy: RoutingStrategy = RoutingStrategy.CAPABILITY,
anthropic_client: anthropic.Anthropic | None = None,
):
self.host = host
self.strategy = strategy
self._client = anthropic_client or anthropic.Anthropic()
# STATIC strategy: pre-built map of tool_name → server_alias
self._static_map: dict[str, str] = {}
# Per-server tool lists for CAPABILITY strategy
self._server_tools: dict[str, list[str]] = {}
def configure_static(self, tool_to_server: dict[str, str]) -> None:
"""Register explicit tool→server mappings for STATIC strategy."""
self._static_map = tool_to_server
def configure_server_tools(self, server_tools: dict[str, list[str]]) -> None:
"""
Register per-server tool names for CAPABILITY strategy.
e.g. {"db": ["search_filings", "get_filing_details"], "ml": ["fuzzy_match_score"]}
"""
self._server_tools = server_tools
def _route_static(self, tool_name: str) -> str:
"""O(1) dict lookup — fastest possible routing."""
# Strip namespace prefix if present
base = tool_name.split("::")[-1] if "::" in tool_name else tool_name
server = self._static_map.get(base)
if server is None:
raise KeyError(f"No static route for tool '{tool_name}'")
return server
def _route_capability(self, tool_name: str) -> str:
"""Iterate per-server tool lists; return first match."""
base = tool_name.split("::")[-1] if "::" in tool_name else tool_name
for server_alias, tools in self._server_tools.items():
if base in tools:
return server_alias
raise KeyError(f"No capability match for tool '{tool_name}'")
async def _classify_intent(self, user_message: str) -> list[str]:
"""
Use a cheap haiku model to classify which servers are relevant.
Returns a list of server aliases, e.g. ["db", "ml"].
WHY: Reduces the tool list shown to the primary model from N to a subset.
"""
server_list = ", ".join(self._server_tools.keys())
prompt = (
f"Which of these MCP servers are needed to answer this request: {server_list}?\n"
f"Request: {user_message}\n"
"Respond with ONLY a JSON array of server names, e.g. [\"db\", \"ml\"]."
)
resp = self._client.messages.create(
model="claude-haiku-4-5",
max_tokens=64,
messages=[{"role": "user", "content": prompt}],
)
import json
text = resp.content[0].text.strip()
# Safely parse; fall back to all servers on parse error
try:
servers = json.loads(text)
return [s for s in servers if s in self._server_tools]
except (json.JSONDecodeError, TypeError):
return list(self._server_tools.keys())
def get_tools_for_strategy(self, servers: list[str] | None = None) -> list[dict]:
"""
Return the tool subset for the given servers (intent strategy).
If servers is None, return full list (static/capability strategies).
"""
if servers is None:
return self.host.get_tools_for_llm()
all_tools = self.host.get_tools_for_llm()
return [
t for t in all_tools
if t["name"].split("::")[0] in servers
]
async def run_agent(
self,
user_message: str,
system_prompt: str = "You are a helpful research agent.",
max_turns: int = 10,
) -> str:
"""
WHAT: Full agent loop using the configured routing strategy.
WHY: Single entry point — caller doesn't need to know which strategy is active.
"""
# Determine tool list based on strategy
if self.strategy == RoutingStrategy.INTENT:
relevant_servers = await self._classify_intent(user_message)
tools = self.get_tools_for_strategy(relevant_servers)
else:
tools = self.get_tools_for_strategy()
messages = [{"role": "user", "content": user_message}]
for _ in range(max_turns):
resp = self._client.messages.create(
model="claude-sonnet-4-5",
max_tokens=4096,
system=system_prompt,
tools=tools,
messages=messages,
)
if resp.stop_reason == "end_turn":
# Extract final text
for block in resp.content:
if hasattr(block, "text"):
return block.text
return ""
if resp.stop_reason != "tool_use":
break
# Process all tool calls in this turn
messages.append({"role": "assistant", "content": resp.content})
tool_results = []
for block in resp.content:
if block.type != "tool_use":
continue
tool_name = block.name # already namespaced: db::search_filings
# Route based on strategy
if self.strategy == RoutingStrategy.STATIC:
alias = self._route_static(tool_name)
call_name = f"{alias}::{tool_name}" if "::" not in tool_name else tool_name
elif self.strategy == RoutingStrategy.CAPABILITY:
alias = self._route_capability(tool_name)
call_name = f"{alias}::{tool_name}" if "::" not in tool_name else tool_name
else:
# INTENT and prefix-based: tool_name is already namespaced
call_name = tool_name
try:
result = await self.host.call_tool(call_name, block.input)
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": result,
})
except Exception as e:
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": f"Error: {e}",
"is_error": True,
})
messages.append({"role": "user", "content": tool_results})
return "Max turns reached without final answer."
# ── Usage ─────────────────────────────────────────────────────────────────
async def main():
host = MCPHost()
await host.add_server("db", "python", ["data_server.py"])
await host.add_server("ml", "python", ["compute_server.py"])
await host.add_server("store", "python", ["storage_server.py"])
router = SmartRouter(host, strategy=RoutingStrategy.INTENT)
router.configure_server_tools({
"db": ["search_filings", "get_filing_details"],
"ml": ["fuzzy_match_score", "calculate_risk_score"],
"store": ["save_result", "recall_similar"],
})
answer = await router.run_agent(
"Find all filings for Acme Corp and calculate their risk score."
)
print(answer)
await host.close()
asyncio.run(main())
/**
* SmartRouter — three configurable routing strategies.
* WHAT: Wraps MCPHost and selects server-specific tool subsets.
* WHY: Reduces tool list noise, improves routing accuracy.
*/
import Anthropic from "@anthropic-ai/sdk";
import type { MCPHost } from "./mcp-host.js";
type RoutingStrategy = "static" | "capability" | "intent";
export class SmartRouter {
private staticMap = new Map<string, string>(); // tool → alias
private serverTools = new Map<string, string[]>(); // alias → tools
constructor(
private host: MCPHost,
private strategy: RoutingStrategy = "capability",
private client = new Anthropic()
) {}
configureStatic(map: Record<string, string>): void {
Object.entries(map).forEach(([k, v]) => this.staticMap.set(k, v));
}
configureServerTools(tools: Record<string, string[]>): void {
Object.entries(tools).forEach(([k, v]) => this.serverTools.set(k, v));
}
private routeStatic(toolName: string): string {
const base = toolName.includes("::") ? toolName.split("::")[1] : toolName;
const server = this.staticMap.get(base);
if (!server) throw new Error(`No static route for '${toolName}'`);
return server;
}
private routeCapability(toolName: string): string {
const base = toolName.includes("::") ? toolName.split("::")[1] : toolName;
for (const [alias, tools] of this.serverTools) {
if (tools.includes(base)) return alias;
}
throw new Error(`No capability match for '${toolName}'`);
}
private async classifyIntent(message: string): Promise<string[]> {
const servers = [...this.serverTools.keys()].join(", ");
const resp = await this.client.messages.create({
model: "claude-haiku-4-5",
max_tokens: 64,
messages: [
{
role: "user",
content:
`Which of these MCP servers are needed: ${servers}?\n` +
`Request: ${message}\n` +
'Respond with ONLY a JSON array, e.g. ["db","ml"].',
},
],
});
const text = (resp.content[0] as { text: string }).text.trim();
try {
const parsed = JSON.parse(text) as string[];
return parsed.filter((s) => this.serverTools.has(s));
} catch {
return [...this.serverTools.keys()];
}
}
private getToolsForServers(servers?: string[]): Anthropic.Tool[] {
const all = this.host.getToolsForLLM();
if (!servers) return all as Anthropic.Tool[];
return all.filter((t) =>
servers.includes(t.name.split("::")[0])
) as Anthropic.Tool[];
}
async runAgent(
userMessage: string,
systemPrompt = "You are a helpful research agent.",
maxTurns = 10
): Promise<string> {
let tools: Anthropic.Tool[];
if (this.strategy === "intent") {
const relevant = await this.classifyIntent(userMessage);
tools = this.getToolsForServers(relevant);
} else {
tools = this.getToolsForServers();
}
const messages: Anthropic.MessageParam[] = [
{ role: "user", content: userMessage },
];
for (let i = 0; i < maxTurns; i++) {
const resp = await this.client.messages.create({
model: "claude-sonnet-4-5",
max_tokens: 4096,
system: systemPrompt,
tools,
messages,
});
if (resp.stop_reason === "end_turn") {
const textBlock = resp.content.find((b) => b.type === "text") as
| { type: "text"; text: string }
| undefined;
return textBlock?.text ?? "";
}
if (resp.stop_reason !== "tool_use") break;
messages.push({ role: "assistant", content: resp.content });
const toolResults: Anthropic.ToolResultBlockParam[] = [];
for (const block of resp.content) {
if (block.type !== "tool_use") continue;
const toolName = block.name;
let callName = toolName;
if (this.strategy === "static") {
const alias = this.routeStatic(toolName);
callName = toolName.includes("::") ? toolName : `${alias}::${toolName}`;
} else if (this.strategy === "capability") {
const alias = this.routeCapability(toolName);
callName = toolName.includes("::") ? toolName : `${alias}::${toolName}`;
}
try {
const result = await this.host.callTool(
callName,
block.input as Record<string, unknown>
);
toolResults.push({ type: "tool_result", tool_use_id: block.id, content: result });
} catch (e) {
toolResults.push({
type: "tool_result",
tool_use_id: block.id,
content: `Error: ${e}`,
is_error: true,
});
}
}
messages.push({ role: "user", content: toolResults });
}
return "Max turns reached.";
}
}
// ── Usage ─────────────────────────────────────────────────────────────────
const host = new MCPHost();
await host.addServer("db", "node", ["data_server.js"]);
await host.addServer("ml", "node", ["compute_server.js"]);
const router = new SmartRouter(host, "intent");
router.configureServerTools({
db: ["search_filings", "get_filing_details"],
ml: ["fuzzy_match_score", "calculate_risk_score"],
});
const answer = await router.runAgent(
"Find Acme Corp filings and score their risk."
);
console.log(answer);
await host.close();
You have three routing strategies: (1) static map — use when tool→server assignments are fixed and you want zero overhead; (2) capability match — use when servers register tools dynamically at startup; (3) intent routing — use when tool list size causes model confusion (typically 30+ tools) and you can afford a fast classifier call.
5. Load Balancing & Failover
Production agents need high availability. A single server instance is a single point of failure — one crash, one deployment, or one network blip takes your agent offline. Load balancingLoad balancing — distributing incoming tool calls across multiple instances of the same server so no single instance is overwhelmed and a failing instance can be bypassed automatically. solves both availability and throughput.
LoadBalancedMCPPool maintains a pool of sessions for the same logical server. Round-robin distributes calls evenly; least-connections routes to the session with fewest in-flight calls. A background health-check loop pings every session every 30 s.
"""
LoadBalancedMCPPool — multiple instances of the same logical server.
WHAT: Round-robin or least-connections across N replicas; auto-failover.
WHY: Zero-downtime deployments, high-throughput parallelism, HA.
"""
import asyncio
import time
from contextlib import AsyncExitStack
from dataclasses import dataclass, field
from enum import Enum
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
class LBStrategy(str, Enum):
ROUND_ROBIN = "round_robin"
LEAST_CONNECTIONS = "least_connections"
@dataclass
class PoolEntry:
session: ClientSession
alias: str
healthy: bool = True
in_flight: int = 0
last_check: float = field(default_factory=time.time)
class LoadBalancedMCPPool:
"""
Pool of identical MCP server instances for HA and throughput.
GOTCHA: All instances must expose identical tool schemas. If they diverge
after a partial upgrade, the LLM may call a tool that doesn't exist on
the routed instance. Use versioned deployments to prevent this.
"""
HEALTH_CHECK_INTERVAL = 30 # seconds
def __init__(self, strategy: LBStrategy = LBStrategy.ROUND_ROBIN):
self.strategy = strategy
self._entries: list[PoolEntry] = []
self._rr_index: int = 0
self._exit_stack = AsyncExitStack()
self._health_task: asyncio.Task | None = None
async def add_instance(
self, command: str, args: list[str], env: dict | None = None, alias: str | None = None
) -> None:
"""Connect a new server instance to the pool."""
params = StdioServerParameters(command=command, args=args, env=env)
read, write = await self._exit_stack.enter_async_context(stdio_client(params))
session = await self._exit_stack.enter_async_context(ClientSession(read, write))
await session.initialize()
idx = len(self._entries)
self._entries.append(PoolEntry(
session=session,
alias=alias or f"instance-{idx}",
))
async def start_health_checks(self) -> None:
"""Launch background health-check loop."""
self._health_task = asyncio.create_task(self._health_loop())
async def _health_loop(self) -> None:
"""Ping each instance every 30 s; mark unhealthy on failure."""
while True:
await asyncio.sleep(self.HEALTH_CHECK_INTERVAL)
for entry in self._entries:
try:
# list_tools() is a lightweight ping — no side effects
await asyncio.wait_for(entry.session.list_tools(), timeout=5.0)
entry.healthy = True
entry.last_check = time.time()
except Exception as e:
entry.healthy = False
print(f"[HealthCheck] {entry.alias} UNHEALTHY: {e}")
def _pick_round_robin(self) -> PoolEntry:
"""
WHAT: Advance index, skip unhealthy, wrap around.
GOTCHA: If all instances are unhealthy, raise immediately — never loop forever.
"""
healthy = [e for e in self._entries if e.healthy]
if not healthy:
raise RuntimeError("All pool instances are unhealthy — no call possible")
self._rr_index = (self._rr_index + 1) % len(healthy)
return healthy[self._rr_index % len(healthy)]
def _pick_least_connections(self) -> PoolEntry:
"""Pick the healthy instance with the fewest in-flight calls."""
healthy = [e for e in self._entries if e.healthy]
if not healthy:
raise RuntimeError("All pool instances are unhealthy")
return min(healthy, key=lambda e: e.in_flight)
def _pick(self) -> PoolEntry:
if self.strategy == LBStrategy.ROUND_ROBIN:
return self._pick_round_robin()
return self._pick_least_connections()
async def call_tool(self, tool_name: str, tool_input: dict) -> str:
"""
WHAT: Pick an instance, increment in-flight counter, call, decrement.
WHY: in_flight tracking enables least-connections strategy.
GOTCHA: Always decrement in-flight even on exception (use try/finally).
"""
entry = self._pick()
entry.in_flight += 1
try:
result = await entry.session.call_tool(tool_name, tool_input)
content = result.content
if content:
return content[0].text if hasattr(content[0], "text") else str(content[0])
return ""
except Exception:
# Mark instance unhealthy on tool call failure
entry.healthy = False
raise
finally:
entry.in_flight = max(0, entry.in_flight - 1)
async def close(self) -> None:
if self._health_task:
self._health_task.cancel()
await self._exit_stack.aclose()
# ── Usage ─────────────────────────────────────────────────────────────────
async def main():
pool = LoadBalancedMCPPool(strategy=LBStrategy.LEAST_CONNECTIONS)
# Start 3 replicas of the same compute server
for port_suffix in range(3):
await pool.add_instance(
"python", ["compute_server.py"],
alias=f"compute-{port_suffix}",
)
await pool.start_health_checks()
# Calls are automatically distributed across healthy instances
results = await asyncio.gather(
pool.call_tool("fuzzy_match_score", {"a": "Acme Inc", "b": "Acme Corp"}),
pool.call_tool("calculate_risk_score", {"entity_id": "E001"}),
pool.call_tool("fuzzy_match_score", {"a": "GlobalTech", "b": "Global Technologies"}),
)
for r in results:
print(r)
await pool.close()
asyncio.run(main())
/**
* LoadBalancedMCPPool — multiple instances of the same logical server.
* WHAT: Round-robin or least-connections; health-check loop; auto-failover.
*/
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";
type LBStrategy = "round_robin" | "least_connections";
interface PoolEntry {
client: Client;
transport: StdioClientTransport;
alias: string;
healthy: boolean;
inFlight: number;
lastCheck: number;
}
export class LoadBalancedMCPPool {
private entries: PoolEntry[] = [];
private rrIndex = 0;
private healthInterval?: ReturnType<typeof setInterval>;
private readonly HEALTH_INTERVAL_MS = 30_000;
constructor(private strategy: LBStrategy = "round_robin") {}
async addInstance(
command: string,
args: string[],
alias?: string,
env?: Record<string, string>
): Promise<void> {
const transport = new StdioClientTransport({ command, args, env });
const client = new Client({ name: "lb-pool", version: "1.0.0" }, {});
await client.connect(transport);
this.entries.push({
client,
transport,
alias: alias ?? `instance-${this.entries.length}`,
healthy: true,
inFlight: 0,
lastCheck: Date.now(),
});
}
startHealthChecks(): void {
this.healthInterval = setInterval(async () => {
for (const entry of this.entries) {
try {
await Promise.race([
entry.client.listTools(),
new Promise((_, rej) => setTimeout(() => rej(new Error("timeout")), 5000)),
]);
entry.healthy = true;
entry.lastCheck = Date.now();
} catch (e) {
entry.healthy = false;
console.warn(`[HealthCheck] ${entry.alias} UNHEALTHY: ${e}`);
}
}
}, this.HEALTH_INTERVAL_MS);
}
private pickRoundRobin(): PoolEntry {
const healthy = this.entries.filter((e) => e.healthy);
if (!healthy.length) throw new Error("All pool instances are unhealthy");
this.rrIndex = (this.rrIndex + 1) % healthy.length;
return healthy[this.rrIndex];
}
private pickLeastConnections(): PoolEntry {
const healthy = this.entries.filter((e) => e.healthy);
if (!healthy.length) throw new Error("All pool instances are unhealthy");
return healthy.reduce((a, b) => (a.inFlight <= b.inFlight ? a : b));
}
private pick(): PoolEntry {
return this.strategy === "round_robin"
? this.pickRoundRobin()
: this.pickLeastConnections();
}
async callTool(
toolName: string,
toolInput: Record<string, unknown>
): Promise<string> {
const entry = this.pick();
entry.inFlight++;
try {
const result = await entry.client.callTool({ name: toolName, arguments: toolInput });
const content = result.content as Array<{ type: string; text?: string }>;
return content.find((c) => c.type === "text")?.text ?? "";
} catch (e) {
entry.healthy = false;
throw e;
} finally {
entry.inFlight = Math.max(0, entry.inFlight - 1);
}
}
async close(): Promise<void> {
clearInterval(this.healthInterval);
for (const { client, transport } of this.entries) {
await client.close();
await transport.close();
}
}
}
// ── Usage ─────────────────────────────────────────────────────────────────
const pool = new LoadBalancedMCPPool("least_connections");
await pool.addInstance("node", ["compute_server.js"], "compute-0");
await pool.addInstance("node", ["compute_server.js"], "compute-1");
await pool.addInstance("node", ["compute_server.js"], "compute-2");
pool.startHealthChecks();
const [r1, r2] = await Promise.all([
pool.callTool("fuzzy_match_score", { a: "Acme Inc", b: "Acme Corp" }),
pool.callTool("calculate_risk_score", { entity_id: "E001" }),
]);
console.log(r1, r2);
await pool.close();
- Three compute server instances were registered in the pool.
- A background loop pings each instance every 30 s and marks it healthy or unhealthy.
- When
call_toolfires, the pool picks the instance with the fewest in-flight calls (least-connections strategy) and increments that counter atomically. - If the call throws, the entry is immediately marked unhealthy and the next call will skip it until the health check restores it.
Failover (skip to next healthy instance) and retry (call the same instance again) are distinct. The pool above implements failover. If your tool calls are idempotent, add retry logic on top with exponential backoff. If they are not idempotent (e.g., send_email), use failover only.
6. Full Stack: Three-Server Agent
This section builds a complete orchestrated system used in the Public Records / UCC Data Engineering domain (Capstone C3). Three independent servers, one host, one agent loop. All code is copy-paste runnable.
DataServer — data_server.py
search_filings (text search over a UCC filing store) and get_filing_details (fetch a single filing by ID). Uses an in-memory dict for demo; swap for a real DB in production.
"""data_server.py — UCC filing search and retrieval."""
import json
from mcp.server.fastmcp import FastMCP
mcp = FastMCP("DataServer")
# Demo data — replace with real DB connection
FILINGS = {
"F001": {"id": "F001", "debtor": "Acme Corp", "secured_party": "First National Bank",
"amount": 500_000, "state": "CA", "status": "active"},
"F002": {"id": "F002", "debtor": "Acme Inc", "secured_party": "Regional Credit Union",
"amount": 125_000, "state": "CA", "status": "active"},
"F003": {"id": "F003", "debtor": "GlobalTech LLC", "secured_party": "Capital Partners",
"amount": 2_000_000, "state": "NY", "status": "terminated"},
}
@mcp.tool()
def search_filings(query: str, state: str = "", max_results: int = 10) -> str:
"""
Search UCC filings by debtor name (case-insensitive substring match).
Returns a JSON list of matching filings with id, debtor, amount, and status.
"""
q = query.lower()
results = [
{"id": f["id"], "debtor": f["debtor"], "amount": f["amount"], "status": f["status"]}
for f in FILINGS.values()
if q in f["debtor"].lower() and (not state or f["state"] == state)
][:max_results]
if not results:
return json.dumps({"results": [], "message": f"No filings found for '{query}'"})
return json.dumps({"results": results, "total": len(results)})
@mcp.tool()
def get_filing_details(filing_id: str) -> str:
"""Retrieve full details for a single UCC filing by its ID."""
filing = FILINGS.get(filing_id)
if not filing:
return json.dumps({"error": f"Filing '{filing_id}' not found"})
return json.dumps(filing)
if __name__ == "__main__":
mcp.run()
/** data_server.ts — UCC filing search and retrieval. */
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
const server = new McpServer({ name: "DataServer", version: "1.0.0" });
const FILINGS: Record<string, { id: string; debtor: string; secured_party: string; amount: number; state: string; status: string }> = {
F001: { id: "F001", debtor: "Acme Corp", secured_party: "First National Bank", amount: 500_000, state: "CA", status: "active" },
F002: { id: "F002", debtor: "Acme Inc", secured_party: "Regional Credit Union", amount: 125_000, state: "CA", status: "active" },
F003: { id: "F003", debtor: "GlobalTech LLC", secured_party: "Capital Partners", amount: 2_000_000, state: "NY", status: "terminated" },
};
server.tool(
"search_filings",
"Search UCC filings by debtor name.",
{ query: z.string(), state: z.string().optional(), max_results: z.number().optional() },
async ({ query, state = "", max_results = 10 }) => {
const q = query.toLowerCase();
const results = Object.values(FILINGS)
.filter((f) => f.debtor.toLowerCase().includes(q) && (!state || f.state === state))
.slice(0, max_results)
.map(({ id, debtor, amount, status }) => ({ id, debtor, amount, status }));
return {
content: [{ type: "text" as const, text: JSON.stringify({ results, total: results.length }) }],
};
}
);
server.tool(
"get_filing_details",
"Retrieve full details for a single UCC filing by ID.",
{ filing_id: z.string() },
async ({ filing_id }) => {
const filing = FILINGS[filing_id];
const text = filing ? JSON.stringify(filing) : JSON.stringify({ error: `Filing '${filing_id}' not found` });
return { content: [{ type: "text" as const, text }] };
}
);
await server.connect(new StdioServerTransport());
ComputeServer — compute_server.py
"""compute_server.py — fuzzy matching and risk scoring."""
import json
from mcp.server.fastmcp import FastMCP
mcp = FastMCP("ComputeServer")
def _levenshtein(a: str, b: str) -> int:
"""Classic edit distance — no external deps."""
if len(a) < len(b):
a, b = b, a
if not b:
return len(a)
prev = list(range(len(b) + 1))
for i, ca in enumerate(a, 1):
curr = [i]
for j, cb in enumerate(b, 1):
curr.append(min(prev[j] + 1, curr[j - 1] + 1, prev[j - 1] + (ca != cb)))
prev = curr
return prev[-1]
@mcp.tool()
def fuzzy_match_score(entity_a: str, entity_b: str) -> str:
"""
Compute a 0-100 similarity score between two entity name strings.
Uses normalized Levenshtein distance. Score 100 = identical, 0 = completely different.
"""
if not entity_a or not entity_b:
return json.dumps({"error": "Both entity_a and entity_b are required", "score": 0})
a, b = entity_a.lower().strip(), entity_b.lower().strip()
max_len = max(len(a), len(b))
if max_len == 0:
return json.dumps({"score": 100, "entity_a": entity_a, "entity_b": entity_b})
dist = _levenshtein(a, b)
score = round((1 - dist / max_len) * 100, 1)
return json.dumps({"score": score, "entity_a": entity_a, "entity_b": entity_b, "edit_distance": dist})
@mcp.tool()
def calculate_risk_score(
amount: float,
status: str,
match_score: float = 100.0,
) -> str:
"""
Calculate a 0-100 lien risk score.
Factors: filing amount, status (active=high risk, terminated=low), entity match confidence.
"""
if amount < 0:
return json.dumps({"error": "amount must be non-negative", "risk_score": None})
# Normalize amount: 0 = $0, 100 = $5M+
amount_score = min(amount / 50_000, 100.0)
status_score = 80.0 if status == "active" else 10.0
# Lower match_score increases uncertainty, reducing risk confidence
confidence = match_score / 100.0
raw = (amount_score * 0.5 + status_score * 0.5) * confidence
risk = round(min(raw, 100.0), 1)
return json.dumps({
"risk_score": risk,
"factors": {"amount_score": amount_score, "status_score": status_score, "confidence": confidence},
})
if __name__ == "__main__":
mcp.run()
/** compute_server.ts — fuzzy matching and risk scoring. */
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
function levenshtein(a: string, b: string): number {
if (a.length < b.length) [a, b] = [b, a];
if (!b.length) return a.length;
let prev = Array.from({ length: b.length + 1 }, (_, i) => i);
for (let i = 1; i <= a.length; i++) {
const curr = [i];
for (let j = 1; j <= b.length; j++) {
curr.push(Math.min(prev[j] + 1, curr[j - 1] + 1, prev[j - 1] + (a[i - 1] !== b[j - 1] ? 1 : 0)));
}
prev = curr;
}
return prev[prev.length - 1];
}
const server = new McpServer({ name: "ComputeServer", version: "1.0.0" });
server.tool(
"fuzzy_match_score",
"0-100 similarity score between two entity names.",
{ entity_a: z.string(), entity_b: z.string() },
async ({ entity_a, entity_b }) => {
const a = entity_a.toLowerCase().trim();
const b = entity_b.toLowerCase().trim();
const maxLen = Math.max(a.length, b.length);
if (!maxLen) return { content: [{ type: "text" as const, text: JSON.stringify({ score: 100 }) }] };
const dist = levenshtein(a, b);
const score = Math.round((1 - dist / maxLen) * 1000) / 10;
return { content: [{ type: "text" as const, text: JSON.stringify({ score, entity_a, entity_b, edit_distance: dist }) }] };
}
);
server.tool(
"calculate_risk_score",
"Calculate a 0-100 lien risk score.",
{ amount: z.number(), status: z.string(), match_score: z.number().optional() },
async ({ amount, status, match_score = 100 }) => {
const amountScore = Math.min(amount / 50_000, 100);
const statusScore = status === "active" ? 80 : 10;
const confidence = match_score / 100;
const risk = Math.min(Math.round((amountScore * 0.5 + statusScore * 0.5) * confidence * 10) / 10, 100);
return { content: [{ type: "text" as const, text: JSON.stringify({ risk_score: risk, factors: { amountScore, statusScore, confidence } }) }] };
}
);
await server.connect(new StdioServerTransport());
StorageServer — storage_server.py
"""
storage_server.py — save and recall analysis results.
In-memory for demo. Replace save/recall with ChromaDB for production vector search.
"""
import json
import time
from mcp.server.fastmcp import FastMCP
mcp = FastMCP("StorageServer")
# In-memory store: {result_id: {entity, risk_score, timestamp, ...}}
RESULTS: dict[str, dict] = {}
@mcp.tool()
def save_result(
entity_name: str,
risk_score: float,
filing_ids: list[str],
notes: str = "",
) -> str:
"""
Persist an entity risk analysis result.
Returns the result_id that can be used for later retrieval.
"""
if not entity_name:
return json.dumps({"error": "entity_name is required"})
result_id = f"R{len(RESULTS) + 1:04d}"
RESULTS[result_id] = {
"result_id": result_id,
"entity_name": entity_name,
"risk_score": risk_score,
"filing_ids": filing_ids,
"notes": notes,
"created_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
}
return json.dumps({"result_id": result_id, "saved": True})
@mcp.tool()
def recall_similar(entity_name: str, min_risk_score: float = 0.0) -> str:
"""
Retrieve previously saved results for entities similar to entity_name.
Filters by optional minimum risk score threshold.
"""
q = entity_name.lower()
matches = [
r for r in RESULTS.values()
if q in r["entity_name"].lower() and r["risk_score"] >= min_risk_score
]
return json.dumps({"matches": matches, "total": len(matches)})
if __name__ == "__main__":
mcp.run()
/** storage_server.ts — save and recall analysis results. */
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
const RESULTS = new Map<string, { result_id: string; entity_name: string; risk_score: number; filing_ids: string[]; notes: string; created_at: string }>();
const server = new McpServer({ name: "StorageServer", version: "1.0.0" });
server.tool(
"save_result",
"Persist an entity risk analysis result.",
{ entity_name: z.string(), risk_score: z.number(), filing_ids: z.array(z.string()), notes: z.string().optional() },
async ({ entity_name, risk_score, filing_ids, notes = "" }) => {
const result_id = `R${String(RESULTS.size + 1).padStart(4, "0")}`;
RESULTS.set(result_id, { result_id, entity_name, risk_score, filing_ids, notes, created_at: new Date().toISOString() });
return { content: [{ type: "text" as const, text: JSON.stringify({ result_id, saved: true }) }] };
}
);
server.tool(
"recall_similar",
"Retrieve saved results for entities similar to entity_name.",
{ entity_name: z.string(), min_risk_score: z.number().optional() },
async ({ entity_name, min_risk_score = 0 }) => {
const q = entity_name.toLowerCase();
const matches = [...RESULTS.values()].filter((r) => r.entity_name.toLowerCase().includes(q) && r.risk_score >= min_risk_score);
return { content: [{ type: "text" as const, text: JSON.stringify({ matches, total: matches.length }) }] };
}
);
await server.connect(new StdioServerTransport());
Putting It All Together — orchestrated_agent.py
"""
orchestrated_agent.py — three-server agent for UCC entity resolution.
WHAT: Connects DataServer + ComputeServer + StorageServer via MCPHost.
Uses SmartRouter with intent routing to minimize tool list per turn.
WHY: The LLM sees only the tools it needs; results are persisted automatically.
"""
import asyncio
# Import MCPHost and SmartRouter from Sections 3 and 4
from mcp_host import MCPHost
from smart_router import SmartRouter, RoutingStrategy
async def main():
host = MCPHost()
# Connect all three servers
await host.add_server("db", "python", ["data_server.py"])
await host.add_server("ml", "python", ["compute_server.py"])
await host.add_server("store", "python", ["storage_server.py"])
# Configure intent router so the classifier knows which tools live where
router = SmartRouter(host, strategy=RoutingStrategy.INTENT)
router.configure_server_tools({
"db": ["search_filings", "get_filing_details"],
"ml": ["fuzzy_match_score", "calculate_risk_score"],
"store": ["save_result", "recall_similar"],
})
system = """You are an expert UCC lien risk analyst.
When given an entity name:
1. Search for filings using db::search_filings
2. For each match, score similarity with ml::fuzzy_match_score
3. Calculate risk with ml::calculate_risk_score
4. Save all high-risk results (score > 60) with store::save_result
5. Check for prior analyses with store::recall_similar
Provide a structured risk summary at the end."""
answer = await router.run_agent(
user_message="Analyze lien risk for 'Acme Corporation' — are there any high-risk filings?",
system_prompt=system,
max_turns=15,
)
print("=== Agent Answer ===")
print(answer)
await host.close()
asyncio.run(main())
/** orchestrated_agent.ts — three-server agent for UCC entity resolution. */
import { MCPHost } from "./mcp-host.js";
import { SmartRouter } from "./smart-router.js";
const host = new MCPHost();
await host.addServer("db", "node", ["data_server.js"]);
await host.addServer("ml", "node", ["compute_server.js"]);
await host.addServer("store", "node", ["storage_server.js"]);
const router = new SmartRouter(host, "intent");
router.configureServerTools({
db: ["search_filings", "get_filing_details"],
ml: ["fuzzy_match_score", "calculate_risk_score"],
store: ["save_result", "recall_similar"],
});
const system = `You are an expert UCC lien risk analyst.
When given an entity name:
1. Search for filings using db::search_filings
2. Score entity similarity with ml::fuzzy_match_score
3. Calculate risk with ml::calculate_risk_score
4. Save high-risk results (score > 60) with store::save_result
5. Check for prior analyses with store::recall_similar
Provide a structured risk summary.`;
const answer = await router.runAgent(
"Analyze lien risk for 'Acme Corporation' — any high-risk filings?",
system,
15
);
console.log("=== Agent Answer ===");
console.log(answer);
await host.close();
- The intent classifier determined the query needed
dbandml(and possiblystore), so those tools were shown first. - Claude called
db::search_filings→ got F001, F002; calledml::fuzzy_match_scorefor each → high similarity on both. - Called
ml::calculate_risk_scorefor each filing; F001 scored 73 (high risk). - Called
store::save_resultfor F001; calledstore::recall_similarto check history. - Returned a structured summary — all within a single
run_agentcall, spanning three physical server processes.
7. Observability for Multi-Server Systems
Distributed tracingDistributed tracing — a technique where a unique trace_id is generated at the start of an agent run and propagated through every tool call across every server. Each server logs the trace_id alongside its own operation logs, so you can reconstruct the full call graph after the fact. is the difference between "something is slow" and "the StorageServer's recall_similar takes 800 ms on entity names with spaces." Without cross-server trace IDs, debugging multi-server agents is guesswork.
TracedMCPHost subclasses MCPHost and injects a trace_id into every tool call. Each server logs the trace_id on receipt. The host collects per-server latency and error data. At the end of a run, get_trace_summary() shows a full call graph.
"""
TracedMCPHost — distributed tracing for multi-server MCP agents.
WHAT: Injects trace_id into every tool call; collects per-server latency.
WHY: Identify bottlenecks, reconstruct call graphs, isolate errors by server.
GOTCHA: trace_id must be a string, not a UUID object, for JSON serialization.
"""
import asyncio
import time
import uuid
from dataclasses import dataclass, field
from typing import Optional
# TracedMCPHost extends MCPHost from Section 3
from mcp_host import MCPHost
@dataclass
class SpanRecord:
trace_id: str
span_id: str
server_alias: str
tool_name: str
start_time: float
end_time: Optional[float] = None
success: bool = True
error: Optional[str] = None
@property
def duration_ms(self) -> float:
if self.end_time is None:
return 0.0
return (self.end_time - self.start_time) * 1000
class TracedMCPHost(MCPHost):
"""
Subclass of MCPHost with distributed tracing.
Each agent run gets a unique trace_id; each tool call gets a span_id.
"""
def __init__(self):
super().__init__()
self._spans: list[SpanRecord] = []
self._current_trace_id: Optional[str] = None
def start_trace(self, trace_id: Optional[str] = None) -> str:
"""Begin a new agent run trace. Returns the trace_id."""
self._current_trace_id = trace_id or str(uuid.uuid4())[:8]
self._spans.clear()
print(f"[Trace] Starting run trace_id={self._current_trace_id}")
return self._current_trace_id
async def call_tool(self, namespaced_name: str, tool_input: dict) -> str:
"""
Override call_tool to inject trace metadata and record latency.
WHY: Every tool call is instrumented without changing individual servers.
"""
if "::" not in namespaced_name:
raise ValueError(f"Tool name '{namespaced_name}' missing namespace prefix")
alias = namespaced_name.split("::")[0]
span_id = str(uuid.uuid4())[:8]
span = SpanRecord(
trace_id=self._current_trace_id or "untraced",
span_id=span_id,
server_alias=alias,
tool_name=namespaced_name,
start_time=time.time(),
)
self._spans.append(span)
# Inject trace metadata — servers that support it can log it
traced_input = {**tool_input, "_trace_id": span.trace_id, "_span_id": span_id}
try:
result = await super().call_tool(namespaced_name, traced_input)
span.end_time = time.time()
print(f" [Span] {namespaced_name} → {span.duration_ms:.1f}ms OK trace={span.trace_id}")
return result
except Exception as e:
span.end_time = time.time()
span.success = False
span.error = str(e)
print(f" [Span] {namespaced_name} → {span.duration_ms:.1f}ms ERR {e} trace={span.trace_id}")
raise
def get_trace_summary(self) -> dict:
"""
Return a full trace summary with per-server latency breakdown.
WHY: One dict to paste into your log aggregator (Datadog, CloudWatch, etc.)
"""
if not self._spans:
return {"trace_id": self._current_trace_id, "spans": [], "total_ms": 0}
total_ms = sum(s.duration_ms for s in self._spans)
by_server: dict[str, dict] = {}
for span in self._spans:
if span.server_alias not in by_server:
by_server[span.server_alias] = {"calls": 0, "total_ms": 0.0, "errors": 0}
by_server[span.server_alias]["calls"] += 1
by_server[span.server_alias]["total_ms"] += span.duration_ms
if not span.success:
by_server[span.server_alias]["errors"] += 1
return {
"trace_id": self._current_trace_id,
"total_calls": len(self._spans),
"total_ms": round(total_ms, 1),
"by_server": {
alias: {
"calls": data["calls"],
"avg_ms": round(data["total_ms"] / data["calls"], 1),
"errors": data["errors"],
}
for alias, data in by_server.items()
},
"spans": [
{
"span_id": s.span_id,
"tool": s.tool_name,
"duration_ms": round(s.duration_ms, 1),
"success": s.success,
"error": s.error,
}
for s in self._spans
],
}
# ── Usage ─────────────────────────────────────────────────────────────────
async def main():
host = TracedMCPHost()
await host.add_server("db", "python", ["data_server.py"])
await host.add_server("ml", "python", ["compute_server.py"])
await host.add_server("store", "python", ["storage_server.py"])
trace_id = host.start_trace()
print(f"Agent run: trace_id={trace_id}")
# Run some tool calls (or use SmartRouter.run_agent with host=traced_host)
await host.call_tool("db::search_filings", {"query": "Acme"})
await host.call_tool("ml::fuzzy_match_score", {"entity_a": "Acme Corp", "entity_b": "Acme Inc"})
await host.call_tool("store::save_result", {"entity_name": "Acme Corp", "risk_score": 73.0, "filing_ids": ["F001"]})
import json
summary = host.get_trace_summary()
print("\n=== Trace Summary ===")
print(json.dumps(summary, indent=2))
await host.close()
asyncio.run(main())
/**
* TracedMCPHost — distributed tracing for multi-server MCP agents.
* WHAT: Injects trace_id into every call; collects per-server latency.
*/
import { MCPHost } from "./mcp-host.js";
import { randomBytes } from "crypto";
interface SpanRecord {
traceId: string;
spanId: string;
serverAlias: string;
toolName: string;
startTime: number;
endTime?: number;
success: boolean;
error?: string;
}
export class TracedMCPHost extends MCPHost {
private spans: SpanRecord[] = [];
private currentTraceId?: string;
startTrace(traceId?: string): string {
this.currentTraceId = traceId ?? randomBytes(4).toString("hex");
this.spans = [];
console.log(`[Trace] Starting run trace_id=${this.currentTraceId}`);
return this.currentTraceId;
}
async callTool(
namespacedName: string,
toolInput: Record<string, unknown>
): Promise<string> {
const alias = namespacedName.split("::")[0];
const spanId = randomBytes(4).toString("hex");
const span: SpanRecord = {
traceId: this.currentTraceId ?? "untraced",
spanId,
serverAlias: alias,
toolName: namespacedName,
startTime: Date.now(),
success: true,
};
this.spans.push(span);
// Inject trace metadata for servers that log it
const tracedInput = { ...toolInput, _trace_id: span.traceId, _span_id: spanId };
try {
const result = await super.callTool(namespacedName, tracedInput);
span.endTime = Date.now();
console.log(` [Span] ${namespacedName} → ${span.endTime - span.startTime}ms OK`);
return result;
} catch (e) {
span.endTime = Date.now();
span.success = false;
span.error = String(e);
console.error(` [Span] ${namespacedName} → ${span.endTime! - span.startTime}ms ERR ${e}`);
throw e;
}
}
getTraceSummary() {
const totalMs = this.spans.reduce(
(acc, s) => acc + ((s.endTime ?? s.startTime) - s.startTime), 0
);
const byServer: Record<string, { calls: number; totalMs: number; errors: number }> = {};
for (const span of this.spans) {
byServer[span.serverAlias] ??= { calls: 0, totalMs: 0, errors: 0 };
byServer[span.serverAlias].calls++;
byServer[span.serverAlias].totalMs += (span.endTime ?? span.startTime) - span.startTime;
if (!span.success) byServer[span.serverAlias].errors++;
}
return {
trace_id: this.currentTraceId,
total_calls: this.spans.length,
total_ms: totalMs,
by_server: Object.fromEntries(
Object.entries(byServer).map(([alias, d]) => [
alias,
{ calls: d.calls, avg_ms: Math.round(d.totalMs / d.calls), errors: d.errors },
])
),
};
}
}
// ── Usage ─────────────────────────────────────────────────────────────────
const host = new TracedMCPHost();
await host.addServer("db", "node", ["data_server.js"]);
await host.addServer("ml", "node", ["compute_server.js"]);
await host.addServer("store", "node", ["storage_server.js"]);
const traceId = host.startTrace();
await host.callTool("db::search_filings", { query: "Acme" });
await host.callTool("ml::fuzzy_match_score", { entity_a: "Acme Corp", entity_b: "Acme Inc" });
console.log(JSON.stringify(host.getTraceSummary(), null, 2));
await host.close();
A production UCC entity resolution agent at a legal data firm averaged 11 tool calls per agent run across 3 servers. Before tracing, debugging an intermittent 4-second slowdown took 2 days. After adding TracedMCPHost, they identified that store::recall_similar was performing a full table scan on entity names with ampersands (&) due to a missing index — 15-minute fix once identified. Distributed tracing converted a guessing game into a 30-second diagnosis.
8. Lab: Build a Three-Server Orchestrated Agent
Follow these steps in order. Each step has a command, expected output, and a checkpoint. If you see the expected output, proceed. If not, check the troubleshooting note.
Start the DataServer
Save data_server.py from Section 6 and verify it starts cleanly.
python data_server.pyLeave this terminal open. The server runs until you close it.
Start the ComputeServer
In a second terminal window, start the compute server.
python compute_server.pyStart the StorageServer
python storage_server.pyCreate MCPHost and connect all three
Save the following as orchestrated_agent.py (the full version from Section 6). Then install dependencies.
pip install mcp anthropicRun an entity resolution query
Verify that tool calls span all three servers in one agent run.
python orchestrated_agent.pyVerify server-level routing in logs
Add TracedMCPHost from Section 7 (replace MCPHost with TracedMCPHost in your agent) and re-run. Verify the trace summary shows calls distributed across all three servers.
Checkpoint: If you see calls distributed across all three servers in the trace summary, your orchestrated multi-server agent is working correctly.
- Server not found: Make sure each
*_server.pyfile is in the same directory asorchestrated_agent.py. - Tool namespace error: Verify
MCPHost.add_server()aliases match yourconfigure_server_tools()keys ("db","ml","store"). - ANTHROPIC_API_KEY missing: Run
export ANTHROPIC_API_KEY=sk-ant-...before starting the agent. - Agent uses wrong server: Check that your intent router alias list matches the actual tool lists. A mismatch causes the classifier to exclude a server.
9. Knowledge Check
Five questions. Each reveals the answer immediately with an explanation.
1. You have 40 tools across 4 servers and your agent frequently calls the wrong server. Which routing strategy addresses this most directly?
2. Two servers both expose a tool named search. Without namespacing, what happens when the LLM calls search?
3. Which load balancing strategy should you use when your tool calls have highly variable latency (some take 10 ms, others 2 seconds)?
4. What is the key advantage of the hierarchical topology over fan-out?
analyze_entity) to the LLM while internally calling multiple sub-servers. This reduces tool list size and hides internal orchestration complexity.5. In TracedMCPHost, why is in_flight decremented in a finally block rather than in the success path only?
finally for any variable decrementfinally runs faster than the success path for numeric operationsfinally is a design choice here, not a language constraint.raise in the try block. Without finally, the counter stays permanently incremented — the least-connections strategy will think the instance is busy forever and route around it, effectively removing it from the pool.