← Starter guides · stop 4 ⬇ PDF
Visual Study Guide · 2026 Edition

AI Agent Engineering

29 modules with diagrams — every concept visualised for fast recall. Sized for the night before a phone screen.
7 Tracks29 ModulesDomain: UCC Lien Risk Pipelineclaude-sonnet-4-6
M00

Course Overview & Agent Fundamentals

Track 1: Foundations
Three levels of LLM programs — who decides what happens next?
Three levels of LLM programs Level 1 LLM Call: your code decides, one request one response. Level 2 LLM Workflow: your code orchestrates fixed sequence. Level 3 Agent: Claude decides at runtime with tools and a loop. Level 1 · LLM Call NOT an agent User question Claude answers Done. YOU decide everything Level 2 · Workflow NOT an agent Step 1: Extract Step 2: Classify Step 3: Report Fixed sequence, YOU choose Level 3 ★ · Agent THIS is an agent Claude reasons loop tool_use → run tool observe result end_turn → done ✓ CLAUDE decides every turn
🧪 Litmus Test

Replace Claude with hardcoded responses. If the program still works — NOT an agent. Level 3 breaks because Claude's runtime decisions can't be hardcoded. That fragility IS the proof.

Same UCC problem — three wrappers, smarter output each lane
Three approaches: Script, FastAPI, Agent Lane 1 Script takes 6 numbers returns probability. Lane 2 FastAPI takes company name returns JSON. Lane 3 Agent takes plain English question returns narrative report with name variant discovery. Lane 1 · Script ~$0 Input: 6 numbers you prep Random Forest model 0.823 · no explanation Misses name variants Model IS the product Lane 2 · FastAPI ~$0 Input: company name Hardcoded ILIKE SQL → ML Rigid JSON · 3 filings found Missed 4 DBA variants New question = new endpoint Lane 3 ★ · Agent ~$0.03 Plain English question Claude reasons + loops Narrative + 7 filings + DBA ✓ Discovers variants by reasoning ML model is ONE tool used
M00

7 Building Blocks of a Production Agent

All Tracks
Drop any block and your agent stops being trustworthy
7 building blocks of a production agent Seven blocks in a row: Brain LLM Track 1, Tools Track 2, Memory Track 3, Plan Track 4, Guardrails Track 5, Eyes Observability Track 6, Home Deploy Track 7. Each stacked on the previous showing cumulative capability. Brain LLM reasoning M01–M04 Without: static rules only Tools APIs & DBs M05–M07 Without: training data only Memory History + RAG M08–M11 Without: amnesic every turn Plan ReAct + decomp M12–M15 Without: one-shot only Guardrails Validate + HITL M16–M18 Without: harmful output, runaway cost Eyes Tracing + logs M19–M20 Without: black box debugging Home FastAPI + Cloud M21–M22 Without: laptop-only demo Most tutorials only teach Brain + Tools — that's 2 of 7. This course covers all seven. Drop any one block → agent stops being trustworthy
M01

LLM Mental Model & Inference

Track 1: Foundations
Two-phase inference: prefill (parallel) then decode (sequential, one token at a time)
Two phases of LLM inference: prefill and decode Phase 1 Prefill: all input tokens processed in parallel, fast, drives time to first token, scales with prompt length, optimized by prompt caching. Phase 2 Decode: one token generated at a time, sequential, slow, drives tokens per second, roughly constant per model. Phase 1 — Prefill All input tokens processed IN PARALLEL tok1 tok2 tok3 … tokN KV cache → TTFT (time to first token) Phase 2 — Decode One token generated at a time — SEQUENTIAL out1 out2 out3 STOP reads KV cache each step → TPS (tokens per second) · roughly constant Prompt caching collapses Phase 1 on repeated prompts → 90% cost reduction on cached prefix
M02

Tokens & Context Window

Track 1: Foundations
200K context window — everything shares one pool. Summarize at 60–70%, not 95%.
Context window budget allocation 200K token window split into: System prompt 1K, Tool definitions 2K, History up to 100K, User message 5K, Response reserve 8K, Safety cushion 84K. Where to cut tokens: output first, then system prompt, then history. Sys Tools History (grows!) Msg Reserve 8K Cushion ~84K 1K 2K up to 100K — summarize at 60–70% 5K 8K max_tokens 200,000 token window (total) Overflow = HTTP 400. No graceful fallback. Manage proactively — never reactively. Output is 3–5× more expensive → cap max_tokens first
M05

Function Calling & The Universal Agent Loop

Track 2: Tool Use
Universal agent loop — every agent is a variation of this
Universal agent loop flowchart User sends question. Claude is called with messages and tools. If stop_reason is end_turn, return final answer. If stop_reason is tool_use, execute each tool, append results, loop back to Claude. User question arrives Pre-call rails: iters · tokens · errors · time CALL Claude messages + tools → response stop_reason? check field end_turn Return answer ✓ tool_use Execute each tool_use block append assistant + tool_results loop back RAISE max iterations exceeded →
Universal Agent Loop pseudocode 
# The universal agent loop — every agent is a variation of this

messages = [user_question]
iters, tokens, errors = 0, 0, 0

WHILE iters < MAX_ITERS AND tokens < BUDGET AND errors < THRESHOLD:

    response = CALL Claude(tools=tool_list, messages=messages)

    messages.append({role: "assistant", content: response.content})
    tokens += response.usage.total

    IF response.stop_reason == "end_turn":           ← happy path ✓
        RETURN final_text(response)

    results = []
    FOR block IN response.content:
        IF block.type == "tool_use":
            out = RUN(block.name, block.input)
            results.append({tool_use_id: block.id, content: out})

    messages.append({role: "user", content: results})   ← BOTH sides required
    iters += 1

RAISE "max iterations exceeded"                        ← safety rail fires
5 stop rails Three guards fire before each Claude call: iteration cap, token budget, consecutive error threshold. Natural completion (end_turn) is the only signal meaning "success" — the others are safety nets for runaway loops.
end_turn Never parse Claude's text to detect completion. stop_reason == "end_turn" is the only reliable signal — scanning natural language for "I'm done" breaks across model versions and prompt changes.
both sides Append both the assistant response and the tool_results before the next API call. Forgetting either causes the API to reject the request. Each tool_result's tool_use_id must exactly match its tool_use's id.
safety rail Start with MAX_ITERS = 10 for most tasks, up to 25 for research agents. A partial result at 8/10 is valuable — return what you have with a note rather than nothing. The rail catches confused loops, not successful completion.
M09

RAG — Retrieval-Augmented Generation

Track 3: Memory & Context
RAG pipeline — ground answers in real documents, not training hallucinations
RAG retrieval augmented generation pipeline Two pipelines: Ingestion pipeline chunks documents, embeds chunks, stores in vector database. Query pipeline embeds user question, finds top-k similar chunks by cosine similarity, injects into system prompt, Claude generates grounded answer. INGESTION (offline) Raw documents Chunk 512–1024 tokens Embed text → vector Vector DB embeddings stored cosine similarity QUERY (online, per request) User question Embed question same model top-k search top-k chunks Inject into system prompt Claude grounded ✓ Answer Hybrid search: dense (semantic embedding) + sparse (BM25 keywords) → re-rank with cross-encoder. Beats pure vector retrieval.
M12

The ReAct Loop

Track 4: Agent Architectures
ReAct: Reason → Act → Observe — the cycle that makes a chatbot an agent
ReAct loop diagram Three repeating phases: Reason (Claude generates thought trace, text block), Act (Claude emits tool_use block, your code runs the tool), Observe (tool result returned as tool_result message). Loop continues until stop_reason end_turn. REASON Claude writes thought trace: text block response.content[].type == "text" ACT Claude emits tool_use. Your code runs the tool. stop_reason == "tool_use" OBSERVE Tool result appended as tool_result msg. messages.append( tool_result) loop until stop_reason == "end_turn" User question → return final answer to user Stop rails: max_iters · token budget · error threshold · wall-clock · end_turn
M14

8 Agent Patterns — Complexity Ladder

Track 4: Agent Architectures
Pick the cheapest pattern that fits the task. Pattern choice swings API cost 10–50×.
8 agent patterns ordered by complexity and cost P1 Single Turn cheapest, P2 ReAct default, P3 Plan Execute, P4 Router, P5 Fan Out, P6 Pipeline, P7 Supervisor Workers, P8 Auto HITL most expensive. Decision tree on left guides which pattern to choose. Decision tree Needs tools? No → just prompt One tool call? Yes → P1 Steps known upfront? Yes → P3 Default → P2 ReAct ★ Input type routes? Yes → P4 Multiple specialists? Yes → P7 High-stakes actions? Yes → P8 P1 · $ Single-Turn Tool Use One question, one tool call, done. Cheapest — always try first. P2 ★ · $$ DEFAULT ReAct Loop Reason → Act → Observe. 2–10 turns. Default agent. P3 · $$ Plan-then-Execute Decompose first, execute in order. Steps known upfront. P4 · $$ Router / Classifier Classify input, dispatch to specialist. 60–70% skip planning. P5 · $$$ Parallel Fan-Out Same task, many inputs in parallel. asyncio.gather. P6 · $$ Pipeline / Chain Sequential stages, each output feeds the next. P7 · $$$ Supervisor + Workers Coordinator delegates to specialist sub-agents. P8 · $$$$ Autonomous + HITL High-stakes: human approval at critical decision points. ↑ Start simple · upgrade only when evidence shows simpler pattern failing ↑ 5 anti-patterns: over-staffing · no max_iters · Plan-Execute for exploratory · fragile fan-out · unvalidated pipeline
M16–M17

Guardrails — Input & Output

Track 5: Guardrails & Safety
5-layer input guardrail stack — cheapest first, fail closed on errors
5-layer input guardrail stack Layer 1 Rate limit 0.1ms free, Layer 2 PII detect redact 1ms free, Layer 3 Injection classifier 200ms $0.002, Layer 4 Schema validation 0ms free, Layer 5 Allow deny log. Blocks on first failure. Fail closed on errors. User message 1 · Rate Limit ~0.1ms · $0 token bucket 2 · PII Detect+Redact ~1ms · $0 regex+NER 3 · Injection Classifier ~200ms · $0.002 clean context 4 · Schema Validation ~0ms · $0 Pydantic/Zod Claude safe ✓ BLOCK on first failure — short-circuit; don't run downstream layers Output validation — 3 checks, cheapest first Claude output Format check $0 · local parse Hallucination check ~$0.003 · RAG verify Toxicity check ~$0.005 · judge User ✓ allow Circuit breaker — 3 states CLOSED normal · count failures N failures OPEN tripped · all → fallback 60s cooldown HALF-OPEN 1 probe request pass → CLOSED fail → OPEN (2× backoff)
M18

Evaluation — Rubrics, Judges & Regression Gates

Track 5: Guardrails & Safety

Agents fail differently from ordinary software, so assertEqual can't test them. It breaks three ways: outputs are non-deterministic, the path the agent takes matters as well as the answer, and quality is a spectrum, not pass/fail. The replacement is rubric scoring on a curated dataset, with statistical thresholds.

The eval pipeline — dataset in, deploy decision out
Agent evaluation pipeline Golden dataset of 50 plus curated cases runs through the agent with full traces, scored on four metrics by a separate judge call, aggregated and stratified by category, compared with a paired t-test, and gated: warn above 5 percent drop, block above 10 percent. Below: an aggregate of 4.2 hides a Shipping category at 2.1, and three CI tiers run at commit, merge and nightly. Golden dataset 50+ curated 60/25/15 mix Run the agent keep full trace tools · tokens Score 4 metrics judge = separate call, 1–5 rubric Aggregate AND by category the truth Paired t-test v_old vs v_new p < 0.05 to ship Deploy gate >5% drop: warn >10%: block AGGREGATE HIDES · CATEGORY SHOWS Aggregate 4.2 Tracking 4.8 Refunds 4.5 Shipping 2.1 Accounts 4.9 "4.2 overall, ship it" masks 5 cases at 2.1 THREE CI TIERS · AFFORDABLE BY DESIGN 1 · Every commit mocked LLM · <30 s · $0 2 · PR merge 10 cases, real API · ~3 min · ~$0.05 3 · Nightly 02:00 UTC 100 cases + full rubric · ~$0.50 Budget cap $300/mo · alert at 80% · pause + page at 100% Full 200-case suite on every commit ≈ $9K/month at 30 PRs/day — that's why the tiers exist.

The rubric — four metrics, stratified by category

MetricWhat it asksHow it's scoredWhat it catches
Task completionDid the agent achieve the stated goal?Binary or partial credit (0–100%) across the suiteThe headline "did this thing work?" number
Tool accuracyRight tools, right parameters?Trace compared with expected tools, fuzzy-matched (parameter order varies)Right answer via the wrong path: brittle, expensive, a time-bomb
Response qualityClarity, completeness, tone, formatting1–5 by Claude-as-judge against a written rubricThe subjective dimension a binary test can't see
EfficiencyTokens, tool calls, latencyLower is better at equal quality4.0 quality at 2K tokens beats 4.2 at 18K for production traffic
Category (tag)Which kind of case was this?Every case tagged, and every metric computed per categoryThe masked failure: the aggregate is for the deck, the category view is for the deploy
Writing the 1–5 levels

Give every score level explicit criteria, e.g. 5 = fully achieved with correct info; 3 = partial, significant gaps. If you can't write the levels, the rubric is too vague to use.

Test it on obviously bad outputs. If those still score 3+, sharpen the level criteria. A useful rubric has a wide score distribution, not everything at 4 or 5.

The judge, in five steps

1) Write a tight rubric. 2) Build the judge prompt: task, output, rubric, and "reason first, then score". 3) Run v1.0 and v1.1 on the same 50–200 cases. 4) Validate the judge against human grades (> 0.7 correlation), re-checked whenever the judge prompt, rubric or model changes. 5) Paired t-test, then decide.

Thresholds to remember

CheckThresholdIf it fails
Judge vs human grades> 0.7 correlationFix the judge prompt or rubric before trusting any score
A/B verdictPaired t-test p < 0.05 and delta > 0"Insufficient evidence": add cases or make a real change. +0.3 on 50 noisy cases is often luck
Inter-annotator agreement> 80% on gold labelsThe rubric is the bug: sharpen it, then relabel
Regression vs baseline> 5% drop warns · > 10% blocksBlock the deploy. A uniform 3% drop on every metric is a red flag too
Flaky caseRetry up to 3×A real failure only if it fails 2 of 3
Dataset mix60% common · 25% edge · 15% adversarialA happy-path-only suite won't detect degradation at the edges
Judge + A/B, and one eval casepseudocode
FUNCTION llm_judge(task, output, rubric):
    prompt = "You are a judge. Score using this rubric.
      TASK: {task}  OUTPUT: {output}  RUBRIC: {rubric}
      REASON FIRST, then return JSON {reasoning, scores}."
    RETURN parse_json(claude.call(prompt))   # SEPARATE call, clean context

FUNCTION ab_compare(v_old, v_new, suite, rubric):
    delta = AVG(scores_new) - AVG(scores_old)
    p     = paired_t_test(scores_old, scores_new)
    IF p < 0.05 AND delta > 0: RETURN "ship v_new"      ← real win
    RETURN "insufficient evidence"

# An eval case stores expected BEHAVIOUR, never an exact output
case = { id: "tc-07",
         input: "Look up 'ACME CORP' vs 'Acme Corporation' — same?",
         expected_behavior: "Recognize entity variants, search both, suggest EIN check",
         category: "entity_resolution", difficulty: "hard", version_added: "v1.3" }
⚠ Three traps

"Aggregate ≥ 4.0 means we're shipping." Read the per-category breakdown first.

"Store exact outputs and assertEqual them." That breaks on day one. Store behaviour and let the rubric judge.

"Build the dataset once." Every production bug and user complaint becomes a new, versioned case. Curated 50 beats random 1,000.

CARDS

30 Interview Flash Cards

Review these before the call
Q: What is a true AI agent? (technical)
A program combining: (1) LLM as reasoning engine, (2) tools to interact with the world, (3) a loop that keeps it thinking and acting until the task is complete. The loop is the entire agent.
Q: Single question separating agents from workflows
"Who decides what happens next — your code, or Claude?" Level 3 = Claude decides at runtime which tool, which args, when to stop. Your code provides tools; Claude provides the brain.
Q: Litmus test for a true agent
Replace LLM with hardcoded responses. If program still works — NOT an agent. Level 3 breaks because Claude's runtime decisions can't be hardcoded. That fragility IS the proof.
Q: Two phases of LLM inference
Prefill: all input tokens processed in parallel; drives TTFT; scales with prompt length. Decode: one token at a time; drives TPS; ~constant per model. Prompt caching collapses repeated prefill.
Q: Default temperature for agents and why
temp=0. At temp 0.8, an agent routing 5K emails/day produces 150–400 unpredictable routing decisions. Reproducibility, debuggability, auditability beat surprise in agentic systems.
Q: Where to cut tokens — priority order
1) Cap max_tokens (output is 3–5× more expensive). 2) Trim system prompt (sent every call). 3) Summarize history at 60–70% utilization. 4) RAG instead of full docs. 5) Tier down to Haiku.
Q: The universal agent pattern (pseudocode)
messages=[question]. WHILE True: response=CALL Claude. IF end_turn → RETURN text. FOR each tool_use → run tool, append result. LOOP. Plus 5 stop rails (iters, tokens, errors, time, natural).
Q: #1 tool-use bug to watch for
Forgetting to append BOTH the assistant response AND the tool_results before the next API call. Also: each tool_result's tool_use_id must exactly match the tool_use's id.
Q: How many tools per request? Why?
Cap at 4–6. Selection accuracy: 97% at 3 tools → 61% at 18 tools. Each schema burns 200–500 input tokens. Beyond 6, split into specialized subagents (M14).
Q: Tool description — what must it include?
WHAT it does (strong verb + concrete object). WHEN to use vs. sibling tools. WHAT it returns (shape/format). Each property also needs its own description with an example value.
Q: Empty result vs error — why does it matter?
{results:[]} = "I checked, nothing matched" → Claude stops. {isError:true} = "I couldn't even check" → Claude asks for help. Conflating causes catastrophic decisions.
Q: MCP vs A2A — one sentence each
MCP = agent↔tool (vertical, Anthropic Nov 2024): how one agent connects to its tools. A2A = agent↔agent (horizontal, Google→Linux Foundation 2025): cross-org delegation with auth. Real systems use BOTH.
Q: 5 stop conditions for a ReAct loop
1) Natural completion (end_turn) — ONLY signal meaning success. 2) Max iterations (5–15). 3) Token/cost budget. 4) Error threshold (2–3 consecutive failures). 5) Wall-clock timeout (30s–2min).
Q: Why thought traces matter
The ACT of generating reasoning text improves Claude's next decision by 20–30%. Don't suppress to save tokens. Do summarize old entries in long sessions. Also provides audit log for compliance.
Q: Plan-Execute vs ReAct — cert exam trap
Plan-Execute: steps can be known before ANY tool runs — brittle when reality differs. ReAct: steps emerge from results — adapts to surprises. Choose by whether steps are knowable before execution.
Q: 8 agent patterns in order of complexity
P1 Single-Turn · P2 ReAct · P3 Plan-Execute · P4 Router · P5 Fan-Out · P6 Pipeline · P7 Supervisor+Workers · P8 Auto+HITL. Pattern choice can swing API bill 10–50× in production.
Q: Subagent vs sub-prompt
A subagent is a separate agent invocation: own system prompt, own small toolset, blank context window, returns structured JSON. Isolation IS the feature. Never let subagents call each other directly.
Q: The coordinator's 4 jobs
1) Route (which subagents). 2) Brief (self-contained task string, resolve pronouns). 3) Collect (structured result, handle errors). 4) Synthesize (one answer). Coordinator owns history; subagents don't.
Q: Input guardrails — 5 layers in order
1) Rate limit (0.1ms, $0). 2) PII detect+redact (regex+Luhn+NER). 3) Injection classifier (separate clean-context Claude call). 4) Schema validation (Pydantic, $0). 5) Allow/deny/log. Fail CLOSED on errors.
Q: Why prompt injection is like SQL injection
Same channel, same vulnerability: data and instructions share one untrusted channel. Fix lives OUTSIDE the model — separate-context classifier on every input AND every tool result (indirect injection).
Q: Output guardrails — 3 checks, cheapest first
1) Format ($0 local parse). 2) Hallucination (~$0.003 RAG-grounded verifier). 3) Toxicity (~$0.005 Claude-as-judge). Skip downstream if upstream blocks. Three verdicts only: allow, redact, block.
Q: 3 HITL gates and when to use each
Approval: irreversible actions. Modification: output needs human voice. Escalation: policy gap/capability limit. Tier on blast radius and reversibility — NOT transaction volume or model confidence score.
Q: Circuit breaker — 3 states
CLOSED (normal) → N failures → OPEN (all→fallback, 60s cooldown) → HALF-OPEN (1 probe) → pass→CLOSED or fail→OPEN with 2× backoff. Different thresholds per failure type.
Q: Why assertEqual fails for agent testing
3 breaks: non-determinism (same prompt → different output), trajectory matters (15 tools for 2-tool job = broken), quality is spectrum not binary. Use rubric scoring on curated golden dataset stratified by category.
Q: 4 evaluation metrics for agents
Task completion rate. Tool accuracy (right tool, right params). Response quality 1–5 (Claude-as-judge). Efficiency (tokens, calls, latency). ALWAYS stratify by category — aggregate hides broken subcategories.
Q: RAG pipeline — 5 steps
1) Ingest (chunk, embed, store). 2) Query (embed question, top-k cosine). 3) Augment (inject chunks into prompt). 4) Generate (Claude answers grounded). 5) Verify (check claims vs sources). Hybrid search beats pure vector.
Q: "Lost in the middle" effect
Claude's attention is strongest at START (system prompt) and END (latest message). Facts buried in a long history get less attention. Put critical instructions at START or END — never in the middle of long docs.
Q: Rate limiting — token bucket mechanics
Bucket holds N tokens; each request consumes 1; refills at R/second. Burst-friendly. Implement per user, per endpoint. Sliding window not fixed window (exploitable boundary). Back with Redis in production.
Q: Observability — print statements vs production
Print statements vanish on restart. Production = structured JSON logs + distributed traces + dashboards. Key: answer "why did Claude pick tool X on request Y at 3:14pm last Tuesday?" with trace_id.
Q: 60-second "whole arc" talking point
"Transformers (2017) understanding. GPT-3 (2020) generating. ChatGPT (2022) mainstream. LLMs alone = chatbots. Agentic AI (2024) adds tool-use, memory, loop, 240× cost drop. I build agents combining ML, RAG, tools, guardrails, observability."
200K
Claude context window — system + tools + history + response share one pool
240×
Cost reduction since GPT-3 2022 ($0.06/1K → $0.00025/1K Claude 3 Haiku)
4–6
Max tools per request (97% selection accuracy at 3 → 61% at 18 tools)
60–70%
Context window utilization trigger for history summarization (never wait for 95%)
3–5×
Output tokens more expensive than input — cap max_tokens first always
20–30%
ReAct (with Reason step) beats action-only on multi-step tasks
Final Takeaway

"An agent is your existing stack with a reasoning layer on top. Same models, same FastAPI, same Docker. New brain. The student who understands the raw loop can learn ANY framework in a day. The student who learned LangChain first often cannot debug without it."

AI Agent Engineering — Visual Study Guide 11 Diagrams · 30 Flash Cards · 7 Tracks claude-sonnet-4-6 · 2026 Edition