How to Structure a Production Claude Agent: All Layers Explained

June 23, 2026 · updated June 25, 2026 · How Claude Actually Works (part 27)

▶ Watch on YouTube & subscribe to The Stack Underflow

Most production Claude agents fail not because the model is bad, but because the system around it has no structure. There is no gateway to absorb traffic spikes, no model router sending cheap steps to a cheap model, no policy hooks guarding destructive tool calls, and no escalation path for the cases the agent cannot handle safely. The result is a fragile, expensive system that hallucinates its way through edge cases and occasionally does something irreversible.

A production agent is not a clever prompt. It is a layered stack where each layer has a specific job and guards a specific failure mode. The good news: every part of the stack is well-understood engineering, and you can build it incrementally. This tutorial walks every layer in order — from the edge of the system to the human handoff — and explains the why behind each one.

The one-sentence version: A production Claude agent is a disciplined stack — gateway, model router, agent loop, scoped tools, policy hooks, structured output boundaries, cached prefix, eval harness, and clean escalation — each layer guarding a distinct failure mode.

The Gateway: Stop Problems Before Any Token Is Spent

The gateway is the first thing a request touches, and it lives entirely outside the model. Its primary job is rate limiting: protecting your cost budget and your upstream dependencies before a single token is processed. It also handles authentication and basic request validation.

The order matters. Authentication, rate limiting, and input filtering are all cheaper to enforce at the edge than inside the loop. A request you block at the gateway costs nothing. A request you block inside the loop has already consumed context tokens, model latency, and potentially triggered tool calls.

In practice the gateway is often a thin layer in front of your agent service — an API gateway, a reverse proxy, or a middleware function. What it must do: reject unauthenticated requests, enforce per-user and per-org rate limits, and validate that the incoming payload is structurally sound before forwarding it.

L0: The Model Router — Smallest Model Per Step

Not every step in an agent’s reasoning is equally hard. A step that formats a date, fetches a row, or calls a well-defined API does not require a frontier model. A step that synthesizes conflicting evidence, reasons across a 200K token context, or makes a judgment call under ambiguity does. Model routing — selecting the right model per step rather than calling the same model everywhere — is one of the highest-leverage cost and latency levers in production agent design.

The current production model family (as of June 2026, per Anthropic’s model overview):

The rule is: smallest model per step, always. Opus 4.8 is 5x more expensive than Haiku on input tokens. A loop that runs 20 steps and routes 16 of them to Haiku and 4 to Opus cuts costs dramatically without sacrificing quality on the steps that actually require it. Model routing is an architectural discipline, not a cost-cutting compromise.

Note: Opus 4.7 and later (including Fable 5) use adaptive thinking via an effort parameter rather than temperature/top-p tuning. Keep this in mind when migrating routing logic from older model families.

L1/L3 Core: The Agent Loop

The agent loop is the heartbeat of the system. Everything else hangs off it. The loop has four steps and they never change:

while not done:
    response = call_model(messages)
    stop_reason = response.stop_reason

    if stop_reason == "tool_use":
        result = run_tool(response.tool_call)
        messages.append(tool_result(result))

    elif stop_reason == "end_turn":
        break

    elif stop_reason == "pause_turn":
        # long-running turn paused — resubmit as-is to continue
        pass

    # policy hook blocks also exit here

Call the model. Read the stop reason. Run a tool if requested. Append the result. Repeat.

The current stop reason values (per Anthropic’s stop reasons docs):

The triad of signals that reconstructs any loop failure in production is: stop reasons, tool calls, and hook blocks. If your logging captures all three on every iteration, you can replay and diagnose any failure.

L2: Reach — Scoped Tools and MCP Servers

The agent touches the world through tools and MCP servers. The key word in a production system is scoped: four or five tools, not forty. A large undifferentiated tool list creates ambiguity for the model, inflates the context window on every call, and expands the policy surface that hooks must guard.

Typical scoped reach in a well-designed agent:

tools:
  - database_query      # read-only, specific schema, no writes
  - web_search          # external lookup, controlled
  - internal_api_call   # one internal service, typed inputs
  - submit_refund       # gated by policy hook (see L3)
  - escalate_to_human   # explicit exit path

MCP (Model Context Protocol) is an open standard (now at 97 million monthly SDK downloads as of March 2026, per modelcontextprotocol.io) for connecting agents to external tools and data sources. It uses a JSON-RPC 2.0 client-server architecture. MCP servers are a natural fit for providing clean, version-controlled interfaces to databases, file systems, and internal APIs — without bloating the tool list with bespoke wrappers.

The 2025-2026 MCP spec added Streamable HTTP transport (replacing SSE), OAuth 2.1 for remote server auth, and stateless operation for horizontal scaling. If you are building an MCP server that will run in production behind a load balancer, use the Streamable HTTP transport.

The architecture of scoped reach looks like this:

         [AGENT LOOP]
               |
         ┌─────┴──────┐
         |             |
    [tools: 4-5]  [MCP servers]
         |             |
    db_query      DB (MCP)
    web_search    internal API (MCP)
    submit_refund
    escalate

    ↑ scope ring: agent sees only what it needs

L3: Control — Policy Hooks and Sub-Agents

Two mechanisms live at this layer:

Pre-tool-use policy hooks. Before the agent is allowed to call a sensitive tool — a database write, a refund path, any irreversible action — a hook gates the call. This is policy enforced in code, not in prompts. Prompt-based guardrails can be eroded by context drift, reasoned around by a sufficiently long conversation, or bypassed by indirect injection. A code-level hook cannot. The hook runs before the tool fires; if it blocks, the tool never executes.

Sub-agent isolation. For high-context work — a long document analysis, a multi-step research pass, a code generation task with large inputs — a sub-agent runs the task in its own isolated context window and returns only a clean, summarized result. The parent loop stays lean. Sub-agents prevent context pollution: the noise from a 150K-token research task does not bleed into the parent’s subsequent decisions. (The Claude Agent SDK, available in Python and TypeScript, ships first-class sub-agent support out of the box.)

[MAIN AGENT LOOP]
       |
  [policy hook] ←— gates submit_refund, db_write, send_email
       |
  [sub-agent bubble]
  ┌─────────────────────────┐
  │  own context window     │
  │  noisy research task    │
  │  returns: clean summary │
  └─────────────────────────┘

Structured Output Boundaries

At every point where the agent crosses a system boundary — calling a tool, returning a result, handing off to another service — output must be schema-forced. The mechanism is a forced tool schema: the model is required to emit a specific JSON structure, not free-form text that gets parsed downstream.

# Fragile: free-form JSON embedded in a string
"result": "{\"status\": \"approved\", \"amount\": 42.00}"

# Correct: schema-forced structured output via tool definition
{
  "status": "approved",
  "amount": 42.00
}

Never free-form JSON in a string. Define the schema in your tool, let the model fill it, and validate on receipt. Classify, extract, and stamp at every boundary — schema enforced. This is what makes agent outputs composable: a downstream system can rely on the shape of the data without defensive parsing logic.

The Cost Plane: Prompt Caching

The system prompt and tool definitions almost never change between calls in an agent loop. Marking them for caching is one of the most impactful single optimizations available. When you add cache_control to stable content blocks, the API caches the computed KV (key-value) state for that prefix. Cache hits are charged at roughly 10% of the standard input token price (per Anthropic’s prompt caching docs).

[SYSTEM PROMPT]         ← mark cache_control here (stable)
[TOOL DEFINITIONS]      ← mark cache_control here (stable)
─────────────────────────────────────────────
[CONVERSATION HISTORY]  ← not cached (changes each turn)
[CURRENT USER MESSAGE]  ← not cached

Practical notes for 2026: caches are now isolated per workspace (as of February 2026), not per organization. A 5-minute cache TTL costs 1.25x the standard write price; a 1-hour TTL costs 2x. The breakeven for the 5-minute window is the second cache hit. For an agent loop running dozens of steps against the same system prompt and tool definitions, this single optimization often makes the difference between a cost structure that scales and one that does not.

The Reliability Plane: Evals and Logging

An offline eval harness gates deploys. Nothing ships if it regresses against a known production set. This is not optional — it is the only way to make incremental changes (prompt wording, model version, tool schema) without flying blind. The eval harness compares agent outputs on a fixed set of production-representative inputs against golden references or an LLM judge. If the pass rate drops, the deploy stops.

Logging strategy in production follows the triad: every loop iteration captures the stop reason, any tool calls made, and any hook blocks triggered. That triad is sufficient to reconstruct the state of the agent at any point in a session and diagnose nearly any failure mode.

[OFFLINE EVAL HARNESS]          [PRODUCTION LOGGING SINK]
     |                                   |
passes? → deploy                  per-iteration:
fails? → block + alert              stop_reason
                                    tool_calls (name + args)
                                    hook_blocks (tool + reason)

Escalation: Clean Exit to a Human

When the agent cannot proceed — due to a policy block, a complexity threshold, an ambiguous risk signal, or an explicit user request — it exits cleanly to a human. The trigger is deterministic, not sentiment-based. “The model feels uncertain” is not a valid escalation trigger; it is subjective and unreliable. A policy gate that fires, a complexity score that crosses a threshold, or a user saying “get me a human” — those are deterministic.

What gets handed off is not a raw transcript. It is a structured summary card: a pre-formatted context object containing the original user intent, the steps the agent completed, the specific escalation reason, and any structured data the human needs to continue. The human reads a summary, not a 40-turn conversation log.

The escalation taxonomy:

The Full Stack at a Glance

[GATEWAY]           auth + rate limits — edge of system
      |
[MODEL ROUTER]      smallest model per step
      |             Haiku 4.5 (cheap) → Sonnet 4.6 → Opus 4.8 (hard)
      |
[AGENT LOOP]        call → stop_reason → tool → append → repeat
      |             log: stop_reason + tool_calls + hook_blocks
      |
[SCOPED REACH]      4-5 tools + MCP servers (Streamable HTTP, OAuth 2.1)
      |
[POLICY HOOKS]      pre-tool-use gates, code-enforced, not prompt-enforced
[SUB-AGENTS]        isolated context for noisy / high-token subtasks
      |
[BOUNDARIES]        schema-forced structured output at every crossing
      |
[CACHED PREFIX]     system prompt + tools marked cache_control (~90% off)
      |
[EVAL HARNESS]      offline regression gate — nothing ships on regression
[LOGGING]           stop_reason + tool_calls + hook_blocks, every iteration
      |
[ESCALATION]        structured summary card → human
                    on: policy / complexity / risk / explicit request
                    never on: sentiment

How to Apply This in Practice

A few concrete steps for teams moving from prototype to production:

1. Add the gateway first. Rate limiting is the cheapest protection you can add. Even a simple token-bucket per user ID prevents runaway costs from bugs or abuse before they reach the model.

2. Audit your tool list. List every tool your agent has. For each one, ask: does this agent’s task genuinely require this? Network access and secret access are the highest-risk categories. Strip anything not essential.

3. Add cache_control to your system prompt and tool definitions. This is a one-line change per content block and typically pays back its cost within the first 2 calls. Check the prompt caching docs for the exact syntax.

4. Write a policy hook before shipping any destructive tool. A destructive tool is any action that cannot be undone: sending, deleting, paying, posting, writing to a database. The hook should be outside the model’s control entirely — a code-level check, not a prompt instruction.

5. Define your escalation triggers before launch. If you cannot state the conditions that trigger escalation as deterministic rules, you do not have an escalation plan — you have hope.

6. Build the eval harness before you need it. The right time to build it is before the first production issue, not after. Start with 20-30 production-representative inputs and a simple pass/fail rubric.

Common Misconceptions

“Prompt guardrails are enough for safety.” Prompts can be eroded by long context, reasoned around by the model, or bypassed entirely by indirect injection (content the agent fetches from the web). Policy hooks enforced in code are the correct mechanism for hard constraints on sensitive tool calls. Code cannot be prompted into ignoring itself.

“More tools means a more capable agent.” Scope is a feature. A large tool list creates model ambiguity (which of these 40 tools should I use?), inflates the context window on every call, and expands the policy surface that hooks must guard. Four to five focused, well-typed tools consistently outperform forty loosely defined ones.

“You should always use the best model.” Routing to the smallest model that can handle each step is architectural discipline. Haiku 4.5 is 5x cheaper than Opus 4.8 on input tokens. A loop that routes routine steps to Haiku and reserves Opus for genuinely hard steps is faster, cheaper, and equally accurate on the hard steps. Using Opus everywhere is not safety — it is laziness.

“Escalation means the agent failed.” Clean escalation is a designed exit path, not a failure mode. An agent that always tries to finish — regardless of complexity, risk, or policy — is an agent that eventually causes an incident. The goal is an agent that knows its own boundary and hands off gracefully when it reaches it.

Frequently Asked Questions

What is the difference between a policy hook and a system prompt instruction? A policy hook is code that runs before a tool call fires, outside the model’s reasoning loop entirely. A system prompt instruction is text that the model reads and attempts to follow — but context drift, long conversations, and adversarial inputs can all erode prompt-based instructions. For hard constraints on sensitive tools, hooks are the right mechanism. Prompts are for behavior guidance, not security boundaries.

How does prompt caching work with the system prompt and tool definitions? You add a cache_control field to stable content blocks in your API request. The API computes and caches the KV state for that prefix. Subsequent requests sharing the same prefix hit the cache at roughly 10% of the standard input cost. The cache is workspace-scoped (as of February 2026). The official prompt caching docs have the exact syntax and TTL options.

When should I use a sub-agent instead of keeping everything in the main loop? Use a sub-agent when a subtask would consume significant context — long document analysis, multi-step research, code generation with large inputs — and the parent loop only needs a clean summary of the result, not every intermediate step. Sub-agents keep the main context window lean and prevent intermediate noise from biasing subsequent decisions. The Claude Agent SDK ships first-class sub-agent support.

What goes into the structured summary card passed to a human on escalation? The summary card should contain: the original user intent, a list of steps the agent completed successfully, the specific escalation reason (policy block / complexity threshold / risk flag / explicit request), and any structured data the human needs to continue. It is not a transcript. It is a pre-digested context object sized for a human to read in 30 seconds and pick up where the agent stopped.

What stop_reason values should I handle in my agent loop? At minimum: end_turn (break the loop), tool_use (run the requested tool and append the result), max_tokens (handle truncation — usually a retry with a pruned context), and pause_turn (resubmit the response as-is to continue a long-running turn). refusal and stop_sequence are less common in agent loops but should not be silently ignored.

How do I decide which model to route to for a given step? A practical heuristic: start with Haiku 4.5 as the default for all steps. Escalate to Sonnet 4.6 for steps that involve multi-hop reasoning, ambiguous classification, or non-trivial synthesis. Reserve Opus 4.8 for steps with genuinely hard long-horizon reasoning — the kind that benefits from the 1M context window or adaptive thinking. Add claude-fable-5 only for the hardest long-horizon steps if Opus is insufficient. Log which model handled which steps in production; the routing data will tell you whether you are over- or under-routing.

Where This Fits in the Series

This tutorial is the capstone of How Claude Actually Works — the episode where every layer built across the series comes together into one system. If this is your entry point, start at The Claude Stack Mental Model for the foundational framing, then read Claude Stack: MCP, Hooks, Skills, Sub-Agents Overview to understand how MCP and sub-agents fit. The Agent Loop deep-dive covers the L1/L3 core in much more detail, and Prompt Caching: Cut Your AI Bill covers the cost plane end to end. The next episode in the series applies this blueprint to a concrete product: Building a Customer Support Agent. Browse all tutorials to follow the full series.