How AI Agents Use Tools: The Model vs. Orchestrator Split

June 23, 2026 · updated June 25, 2026 · AI Agent Internals: How Coding Agents Really Work (part 1)

▶ Watch on YouTube & subscribe to The Stack Underflow

You ask an AI coding agent to read a file, edit some code, and run the build. It does all three. But here is the part that surprises most developers the first time they really sit with it: the language model inside that agent never touched your file system. It never executed a shell command. It never called an API. All it did was write text.

Understanding this split — between what the model does and what the orchestrator does — is the single most load-bearing concept in all of AI agent engineering. Every framework you will ever use (LangGraph, CrewAI, Claude Code, AutoGen, raw Anthropic SDK), every protocol you will ever wire up (MCP, A2A, OpenAI function calling), and every production failure you will ever debug traces back to this seam. The model reasons. The orchestrator acts. Nothing crosses that boundary without your code in the middle.

The one-sentence version: A language model can only produce text, so when an agent “uses a tool,” the model emits a structured text object describing what it wants, and a separate piece of software called the orchestrator actually executes the action and hands the result back as more text.

The model is a text transformer, nothing more

A language model has one job: take a sequence of tokens as input, produce a probability distribution over the next token, repeat until done. That is its entire interface with the world. It cannot open a file descriptor. It cannot make an HTTP request. It cannot fork a process. Inference is a forward pass through a neural network — there is no syscall in there, and there never will be.

This is not a temporary limitation waiting to be fixed by the next model release. It is the architecture. When Claude Opus 4.8 or Sonnet 4.6 appears to “run code,” what is actually happening is that the model emitted text describing code to run, and something else — your orchestrator — ran it. The model’s participation ended the moment the tokens left the sampling loop.

The implication is architectural: every capability an AI agent appears to have is layered on top of, and despite, this constraint. The richer the agent, the more engineering lives in the orchestrator.

How tool schemas enter the context window

Before any agentic loop begins, the orchestrator constructs the model’s input. Tool definitions are part of that input. Each tool is described by a structured schema — JSON Schema in the Claude and OpenAI APIs — specifying the tool name, a description the model uses to decide when to call it, and the argument types the orchestrator expects to receive back.

{
  "name": "read_file",
  "description": "Read the full contents of a file at the given path",
  "input_schema": {
    "type": "object",
    "properties": {
      "path": {
        "type": "string",
        "description": "Absolute path to the file"
      }
    },
    "required": ["path"]
  }
}

The model reads this schema the same way it reads any other text in its context window — left to right, as tokens, with no special register or privileged memory. It now understands: “If I want to read a file, I should emit output that matches this contract.” The description field is not metadata — it is part of the model’s reasoning input, and writing it well is one of the highest-leverage prompt-engineering decisions you will make.

As of mid-2026, tool schema tokens add a fixed overhead per request. Claude Sonnet 4.6 and Haiku 4.5 consume roughly 497 tokens for the tool-use system prompt (with tool_choice: auto) plus the token count of the schema definitions themselves (docs.anthropic.com, 2026). That overhead is paid on every turn of the loop — a cost driver worth understanding before you reach the economics section.

The tool-call loop, step by step

ORCHESTRATOR
============

1. Build context:            4. Execute actual action:
   system prompt +              open file / run shell /
   tool schemas +               call API / query DB
   conversation history
        |                              |
        v                             v
   +----------+  stop_reason:   +-------------------+
   |  MODEL   |  "tool_use"     |  Parse & validate |
   | (LLM)    | --------------> |  tool_use block   |
   +----------+                 +-------------------+
        ^                              |
        |   tool_result               |
        +------------------------------+

5. Append tool_result to
   conversation, loop back
   to step 1

Here is what each step means in concrete API terms (using the Claude Messages API, but the structure is identical across providers):

Step 1 — The orchestrator sends a request. The tools array carries the schema definitions. The messages array carries the conversation so far.

Step 2 — The model decides it needs a tool. Instead of producing prose, the model sets stop_reason: "tool_use" and includes one or more tool_use content blocks in its response. Each block has a unique id, the tool name, and an input object of arguments.

Step 3 — The orchestrator catches the signal. It reads stop_reason, extracts the tool_use blocks, and validates the arguments against the schema. The model is now idle — it is waiting for your code to do something.

Step 4 — The orchestrator executes the real action. This is the only moment anything actually happens in the physical world. A file gets opened. A shell command fires. An HTTP request leaves the process. The model is completely uninvolved.

Step 5 — The result re-enters as tokens. The orchestrator formats the result as a tool_result content block, appended to the conversation as a user-role message. From the model’s perspective, it just received more input tokens. It has no awareness that time passed or that external systems were invoked.

Step 6 — The loop continues until end_turn. The model reads the result, reasons about it, and either calls another tool or produces a final response. The orchestrator’s while stop_reason == "tool_use" loop drives this until stop_reason becomes end_turn, max_tokens, stop_sequence, or refusal.

Parallel tool calling and the token economics of loops

Modern models — including Claude Sonnet 4.6, Opus 4.8, and Haiku 4.5 — can return multiple tool_use blocks in a single response (docs.anthropic.com, 2026). This is parallel tool calling: the model has determined that several actions can be taken simultaneously and emits them all at once. A well-designed orchestrator fires them concurrently, collects all results, and returns them together in the next turn as an array of tool_result blocks. One fewer round trip, meaningfully lower latency.

The token economics of agentic loops are worth internalizing before you reach production:

Re-sent context is the dominant cost driver because every loop turn re-transmits all prior messages as input tokens. A 5-step loop runs roughly 3x the token cost of a single call. A 50-step loop can run 30x. A 200-step autonomous session can hit 100x (LeanOps, 2026). This is not a bug — it is a structural consequence of how stateless inference works: the model has no persistent memory, so the orchestrator must supply the entire conversation on every turn.

The most effective mitigation is prompt caching: Anthropic’s cache-read pricing charges roughly 10% of the base input rate for cached tokens. On a 3,000-token system prompt across a 50-step loop, caching saves approximately 88% of the system prompt cost (Stevens Online / LeanOps analysis, 2026). Tool schemas that do not change between turns are equally cacheable. See Prompt Caching: Anthropic vs OpenAI for the full mechanics.

Everything consequential lives in the orchestrator

Because the model only writes text, every decision about what is allowed to happen sits in the orchestrator. This is where permissions, sandboxing, safety filtering, and cost controls live — not in the model weights.

The practical consequence: when you ask “why does this agent need filesystem access?” or “how does Claude Code decide whether to run a destructive command?”, the answer is always in the orchestrator, not in the weights. The model cannot grant itself permissions. It can only ask, and the orchestrator decides whether to comply.

MCP: the standardised tool wire

The Model Context Protocol (MCP) is an open protocol that standardises how tools are described, discovered, and invoked — a USB-C adapter for the orchestrator layer. Instead of every tool provider writing custom glue code for every orchestrator, MCP defines a shared JSON-RPC 2.0 message format that any host and server can speak.

Host (your orchestrator)
        |
   MCP Client
        |
  JSON-RPC 2.0 over
  Streamable HTTP / stdio
        |
   MCP Server
  (file system, DB, API, ...)
        |
  Tools / Resources / Prompts

The current stable spec is 2025-11-25 (modelcontextprotocol.io). The 2026 roadmap focuses on four areas: stateless HTTP transport (so MCP servers can scale horizontally without sticky sessions), a Tasks extension for long-running operations with retry semantics, enterprise auth aligned with OAuth/OIDC, and a Working Group governance model to reduce review bottlenecks (MCP Blog, 2026). An MCP-capable orchestrator can connect to any MCP server without custom integration — the protocol handles capability negotiation, tool schema exchange, and result serialisation.

From the model’s perspective, MCP tools look identical to native function-calling tools: they appear as schema definitions in the context window, and results return as tool_result tokens. The protocol is an orchestrator concern, invisible to the model.

Observability: tracing the loop with OpenTelemetry

When an agent uses tools across multiple turns, understanding what happened requires structured telemetry. The OpenTelemetry GenAI semantic conventions (currently experimental, opentelemetry.io/blog/2025/ai-agent-observability/) define a standard schema for spans that cover LLM calls, tool calls, and agent reasoning steps. Key attributes include gen_ai.operation.name (e.g., chat, tool_use), gen_ai.system (e.g., anthropic), gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and per-tool-call spans that record the tool name, input, output, and latency.

Instrumentation belongs in the orchestrator’s loop — wrap each model call and each tool execution in a span. The result is a trace tree showing exactly which tool was called, with which arguments, how long it took, and how many tokens each turn consumed. By 2026, OpenTelemetry’s GenAI conventions have been adopted by Datadog, Google Cloud, AWS, and Azure, making cross-platform agent tracing practically feasible (Datadog Engineering, 2025). See Agent Observability for a deeper treatment.

How to apply this right now

Concrete, ordered steps:

1. Make the orchestrator/model boundary explicit in your code. The function that calls the model API, the function that executes tools, and the loop that drives them should be clearly separated. If they are tangled in the same function, you will struggle to add validation, cost tracking, or safety gates later.

2. Write tool descriptions as if you are writing documentation for a developer who knows nothing about your system. The model decides whether to call a tool based almost entirely on its description. Vague descriptions produce incorrect calls; precise descriptions produce reliable ones. Add strict: true to your tool definitions (supported on Claude Sonnet 4.6 and later) to guarantee schema conformance on every call.

3. Enable prompt caching for your system prompt and static tool schemas. This is the highest-ROI single change you can make to agent economics. The tool schema overhead (497-589 tokens per turn on current Claude models) and system prompt are paid on every loop iteration — cache them.

4. Use parallel tool calling where the model supports it. Claude Opus 4.8, Sonnet 4.6, and Haiku 4.5 all emit parallel tool_use blocks when tasks are independent. Make sure your orchestrator executes them concurrently rather than serially — the latency savings compound over long loops.

5. Add a pre-tool-use gate for any irreversible action. Before your orchestrator fires a shell command, writes a file, or sends a network request, run a deterministic check. The model cannot override code you wrote. This is where safety lives.

6. Instrument every loop turn with OpenTelemetry spans. Token counts, tool latencies, and loop depth are the three metrics you need to understand cost and reliability in production. Add them before you need them.

Common misconceptions

“The model is executing code.” It is not. The model generates text that describes code to execute. Execution is handled by the orchestrator, which may or may not honour the request depending on its permission model. A model with a run_shell tool can be sandboxed to a read-only container just as easily as it can be given root access — that decision belongs entirely to the orchestrator.

“Tool calls are a special API feature, not real text.” They feel special in SDKs, but at the token level they are structured text the model learned to produce through training. The API gives them special treatment by parsing them into tool_use blocks, but the model generates them the same way it generates any other structured output.

“The model sees the file system.” The model sees the contents of files only after the orchestrator has fetched them, serialised them to text, and inserted them into the context window as tool_result tokens. It never has a live view of anything. Between loop turns, it has no state at all.

“Server-executed tools are different in kind.” Anthropic runs server-side tools (web_search, code_execution, web_fetch) on its own infrastructure, which means the loop runs inside Anthropic’s servers rather than your application. But the architecture is the same: a model, an orchestrator, and results that re-enter the context as tokens. The stop_reason: "pause_turn" signal on server tools is just a visibility window into that internal loop.

Frequently asked questions

Why does the model emit JSON for tool calls rather than plain English?

JSON is unambiguous and machine-parseable. The orchestrator needs to extract the tool name and arguments reliably at scale. Plain English extraction is a brittle, error-prone parsing problem — a JSON schema gives the model and the orchestrator a shared, validated contract. The id field on each tool_use block lets the orchestrator match results to calls when parallel tool calls are in flight simultaneously.

What happens if the model produces a malformed tool call?

The orchestrator catches the schema validation error before any execution occurs. Depending on the implementation, it may retry by re-prompting the model with the error message appended as a tool_result, surface the error to the user, or apply fallback logic. The model does not know the call failed until the orchestrator tells it in the next turn’s tokens. Adding strict: true to tool definitions (docs.anthropic.com, 2026) eliminates the malformed-call failure mode at the cost of slightly higher refusal rates when the model is uncertain.

Do all agents — Cursor, Copilot, Claude Code — work this way?

Yes. The model-orchestrator architecture is universal. Every agent that appears to take real-world actions is doing it through some form of this loop. The differences between agents are entirely in the orchestrator: which tools are exposed, what the permission model looks like, how results are chunked back into context, which protocol wires tools in (native function calling, MCP, or something custom), and what safety gates sit in front of irreversible actions.

What is MCP and how does it fit the model-orchestrator picture?

MCP (Model Context Protocol, spec version 2025-11-25) is a standardised JSON-RPC 2.0 wire format that lets any tool provider plug into any orchestrator. The orchestrator acts as an MCP host; it connects to MCP servers that expose tools as structured schemas. Those schemas are injected into the model’s context window exactly like native tool definitions. The model emits tool_use blocks; the orchestrator routes them to the appropriate MCP server; results come back as tool_result tokens. MCP is an orchestrator-layer protocol — the model never knows whether a tool is native or MCP-backed.

How much more do agentic loops cost than single-turn calls?

Substantially more. Re-sent context alone is 62% of a typical agent’s token bill (LeanOps, 2026). A 5-step loop runs roughly 3x the token cost of a single call; a 50-step loop can run 30x; a 200-step autonomous session can hit 100x. The mitigation is prompt caching (88% reduction on cached system prompts) and parallel tool calling (fewer round trips). The economics are covered in depth in Why AI Coding Bills Explode.

How do I observe what my agent is actually doing?

Instrument your orchestrator loop with OpenTelemetry spans using the GenAI semantic conventions (opentelemetry.io, 2025). Wrap each model call in a gen_ai.chat span and each tool execution in a child span. Record gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, tool name, arguments (sanitised), and latency on every turn. This trace tree will answer the two most important production questions: “why did it call that tool?” and “why did it cost that much?” The full observability architecture is in Agent Observability.

Where this fits in the series

This episode is the starting point for the “AI Agent Internals: How Coding Agents Really Work” series. The model-orchestrator split introduced here is the foundation every subsequent episode builds on:

• What is MCP: Model Context Protocol — how the orchestrator layer gets standardised tool wiring across any provider
• MCP: The Universal Adapter — the protocol mechanics in detail: JSON-RPC transport, capability negotiation, the 2025-11-25 spec
• The Choice That Defines AI Coding Agents — how agents decide which tools to expose and what the permission model looks like
• Agent Observability — instrumenting the loop so you can see what is happening inside it

For the cost side of the model-orchestrator loop, continue with Why AI Coding Bills Explode and Context Rot Explained. Browse all tutorials to follow the full series.