Agent Observability: How to Trace and Debug AI Agents with OpenTelemetry

June 23, 2026 · updated June 25, 2026 · Agents at Scale: The 2026 Frontier (part 5)

▶ Watch on YouTube & subscribe to The Stack Underflow

Every agent team hits the same wall eventually. The dashboard is green. Every LLM call returned 200. Every tool executed without throwing an exception. Latency was nominal. And the user is staring at output that is confidently, completely wrong.

This happens because traditional monitoring is built on a contract that breaks completely in agentic systems: if the request returned 200 and latency was in budget, everything is fine. A model can produce a well-formed, grammatically perfect answer that is factually wrong. A tool can execute successfully with the wrong arguments. A handoff between agents can pass corrupted state, and both sides will report success. In agent systems, correctness — whether the agent did the right thing — is entirely orthogonal to success — whether the infrastructure did not error. Logs and status codes measure the latter. You need traces to measure the former.

The one-sentence version: Status codes tell you whether requests completed; traces tell you whether your agent did the right thing — and in agentic systems, those two things are almost never the same.

The Field Has a Standard Now

Before 2025, every observability vendor had its own proprietary schema for LLM spans. You picked a vendor, wrote to their SDK, and were locked in. In 2025-2026, that changed. The OpenTelemetry GenAI Special Interest Group — which has been developing the GenAI Semantic Conventions since April 2024 — shipped a standard that the entire industry has now converged on (opentelemetry.io/docs/specs/semconv/gen-ai/, 2026).

The spec is organized in layers, and their stability differs:

“Experimental” here does not mean untested — the agent and framework spans have been stable in practice through 2026 Q1, and major frameworks emit them reliably (callsphere.ai, 2026). It means the OTel governance process has not yet cut a stable release for those layers. Use them; just pin your instrumentation library version and watch the changelog.

The payoff of adopting the standard is real: one span vocabulary works across Datadog, Honeycomb, Grafana, New Relic, Langfuse, MLflow, Braintrust, Arize Phoenix, and any other OTel-compatible backend. Datadog added native support for OTel GenAI conventions at v1.37 in December 2025 (datadoghq.com/blog/llm-otel-semantic-convention/). You instrument once; you query everywhere.

The Six-Layer Span Hierarchy

A production agent trace is not a flat list of LLM calls. It is a nested tree organized across six conceptual layers. Understanding the nesting is what lets you read a trace and immediately know where the failure happened.

invoke_workflow (INTERNAL)          # Multi-agent orchestration root
  invoke_agent: orchestrator (INTERNAL)   # One agent's full run
    chat anthropic (CLIENT)               # LLM call — model makes a plan
    execute_tool: search_kb (INTERNAL)    # Tool call with args + result
    chat anthropic (CLIENT)               # LLM call — model evaluates result
    agent.handoff: sub_agent_A            # State passed to next agent
      invoke_agent: sub_agent_A (INTERNAL)
        chat anthropic (CLIENT)
        execute_tool: send_email (INTERNAL)

Every span has a gen_ai.operation.name attribute that tells you its role. The span name follows a human-readable convention: {operation} {provider} for model calls (e.g., chat anthropic, chat openai), and execute_tool {tool_name} for tool spans (e.g., execute_tool search_kb). Span kind encodes whether the call crosses a process boundary: CLIENT for remote calls (hitting the Anthropic API), INTERNAL for in-process execution (an agent reasoning loop running in your process).

The six layers defined by the spec:

The MCP layer deserves special attention. When an agent calls a tool over MCP, you get a client/server span pair linked by W3C traceparent/tracestate headers. The MCP client-side span carries mcp.method.name, mcp.session.id, mcp.protocol.version, and gen_ai.tool.name. Critically, MCP instrumentation enriches the existing execute_tool span rather than creating a duplicate — you do not get two tool spans in the tree, you get one that now has both the semantic tool data and the protocol-level detail (greptime.com, May 2026).

The Rule That Separates Teams Who Debug from Teams Who Guess

Here is the insight that changes how you think about instrumentation: the most informative span is almost never the LLM call.

LLM calls almost always succeed at the protocol level. The model returns a well-formed response. The bug lives elsewhere:

Where bugs actually live in agent traces
-----------------------------------------
LLM call span         --> Almost always 200. Rarely the bug.
  |
  +-- execute_tool    --> Arguments matter. Wrong args = "succeeded" badly.
  |     |
  |     +-- return    --> Unexpected shape? Model misinterprets it silently.
  |
  +-- agent.handoff   --> State crossing here. This is where truth lives.
        |
        +-- corrupted state  --> Both sides report success. Output is wrong.

The debugging hierarchy, ordered by diagnostic signal per span:

1. Handoff spans — what state did agent A hand agent B? If this is wrong or missing, everything downstream is wrong regardless of what the LLM produces.
2. Tool I/O spans — what arguments did the model pass, and what came back? A hallucinated argument to a real tool produces a real (wrong) result. The tool “succeeds.”
3. State mutation spans — what was in working memory before and after each step? Context drift and summarization loss show up here.
4. LLM call spans — useful for token cost, latency, and finish reason. Rarely where the root cause lives.

Braintrust’s 2026 agent observability guide makes this concrete: each execute_tool span should capture the tool name, the full argument payload, the raw return value, duration, retry count, and error state. If you omit arguments and return values from tool spans, you have structured logs, not a trace — you cannot reconstruct what the agent actually did.

What to Actually Instrument: Attributes That Matter

For a production agent trace, these are the attributes you must capture. The first column uses the official OTel GenAI attribute names.

On every LLM call span:

On every tool execution span:

On every agent invocation span:

The gen_ai.response.finish_reasons attribute is consistently undervalued. A max_tokens finish reason means the model was truncated mid-thought — the response is incomplete, and downstream agents will be working from a partial output. Without this attribute in your spans, truncation is invisible.

Capturing Content Without Burning Down Compliance

By default, the OTel GenAI spec does not record prompt or completion content in span attributes. Content is captured as span events — gen_ai.content.prompt and gen_ai.content.completion — and those events are separate from the span lifecycle. This design is deliberate: you can suppress content events in production (for GDPR/CCPA compliance) without touching the span structure that feeds your latency and cost dashboards (zylos.ai, February 2026).

Three-layer protection strategy, applied in order:

Layer 1: SDK — set capture_message_content = false
           Content events never enter the pipeline.

Layer 2: OTel Collector processor — redaction rule
           Catches anything that slips through Layer 1.
           Runs before data leaves your network.

Layer 3: Backend access control — RBAC + audit log
           Assumes Layers 1 and 2 each fail at least once.

Datadog’s guidance (datadoghq.com, December 2025) emphasizes that the OTel Collector is the right enforcement point for redaction and sampling policies: “you can apply processors for redaction, sampling, enrichment, and routing so your data policies are enforced before telemetry data leaves your network.” This is also the place to handle the external-storage pattern: for systems that must retain full prompt/completion content for debugging, write the content to S3 or your own storage, and put a reference URL on the span instead of the raw content. Trace structure stays intact; customer data never hits the observability SaaS.

Sampling Strategy: What to Keep, What to Drop

You cannot store 100% of every trace in production at scale, and you should not try. The recommended tail-based sampling policy buffers spans and decides after the full trace completes:

Tail-based sampling policy (2026 production baseline)
-------------------------------------------------------
Keep 100%  of traces that contain any ERROR span
Keep 100%  of traces with duration > 5 seconds
Keep 100%  of traces with > 20 spans (high-complexity proxy)
Keep  5%   of remaining successful traces (routine baseline)
Never sample away handoff or tool I/O spans within kept traces

Tail-based sampling is key here — head-based sampling (deciding at trace start) cannot know if a trace will error or become interesting. You need to buffer the full trace, evaluate it, then decide (zylos.ai, February 2026).

The “never sample away handoff or tool I/O spans within kept traces” rule is critical. If you keep a trace but randomly drop its tool execution and handoff spans, you have kept the skeleton and thrown away the bones that break. When you sample a trace, keep it fully.

Two metrics are mandatory regardless of sampling: gen_ai.client.token.usage (histogram, exponential buckets) and gen_ai.client.operation.duration (histogram, seconds). These are aggregated, not sampled — they give you cost and latency baselines even when 95% of traces are dropped.

How to Apply This Right Now

Concrete steps, ordered by impact and speed of implementation:

1. Add auto-instrumentation first. OpenLLMetry patches your LLM SDK client before your first span. MLflow auto-instruments 60+ frameworks including LangChain, LangGraph, CrewAI, OpenAI Agents SDK, LlamaIndex, DSPy, PydanticAI, and Google ADK with a single call. If you are using any of these frameworks, you can have OTel-compliant spans in your backend within an hour (mlflow.org, 2026).

2. Manually instrument your handoffs. Auto-instrumentation covers LLM calls and tool execution. The handoff spans — what state agent A sends agent B — are the highest-signal spans in multi-agent systems and are almost never covered automatically. Write these by hand. Capture: sending agent identity, receiving agent identity, and a snapshot of the state being passed. A content hash or structured summary is fine if the payload is large.

3. Add gen_ai.agent.outcome to every invoke_agent span. The three values — success, error, human_handoff — let you split token spend by outcome category. If your agents are spending 40% of their tokens on error-recovery loops, that shows up in dashboards once you have this attribute.

4. Set up tail-based sampling before you generate volume. It is much harder to implement retroactively. If you are on the OTel Collector stack, the tail_sampling processor is built in.

5. Suppress content events in production from day one. Turn on capture_message_content = false at the SDK level and add a collector redaction processor before your first deployment. It is far easier to relax privacy controls later than to remediate a compliance incident.

6. Add stuck-loop detection. Count execute_tool child spans per invoke_agent span. If an agent is calling the same tool more than N times in one run, emit a metric and alert. Tool-call loops that never resolve are a common silent failure mode — the agent “runs” but is spinning, burning tokens and not converging.

Common Misconceptions

“A green dashboard means my agent is working correctly.” Green metrics mean requests completed without infrastructure errors. Correctness — whether the agent did the right thing with the right information — is not captured by any health check, status code, or latency metric. You need step-level traces to evaluate correctness, and even then you need automated eval gates to score outputs systematically.

“Instrumenting LLM calls is enough for agent observability.” LLM call spans are the least informative spans when debugging agent failures. The tool I/O spans and handoff spans are where the actual root cause almost always lives. LLM call spans are essential for cost and latency; they are rarely the failure point.

“I need a custom observability layer for agents — OTel was built for microservices.” The OTel GenAI Semantic Conventions exist precisely because the GenAI SIG recognized that agent execution is not like microservice tracing. invoke_agent, invoke_workflow, and execute_tool are agent-native constructs. The spec covers the execution tree structure that makes agents legible. Use it; do not reinvent it.

“Capturing full message content in traces is fine for debugging.” It is fine in a local dev environment. In production, raw prompt and completion content in your observability backend creates GDPR/CCPA exposure, potential credential leakage (if prompts include API keys or tokens), and vendor compliance risk. Apply the three-layer strategy — SDK suppression, collector redaction, RBAC — before the first production deployment. OWASP elevated Sensitive Information Disclosure to LLM02 in the 2025 Top Ten specifically because agentic systems now expose far more organizational data through their context windows than chat interfaces did.

Frequently Asked Questions

What is the actual stability status of the OTel GenAI spec right now?

As of mid-2026, the spec is split. Client spans (the chat and text_completion spans for LLM calls) and client metrics reached stable status in early 2026. Agent and framework spans (invoke_agent, invoke_workflow, execute_tool) are still experimental but have been stable in practice through Q1 2026 — major frameworks emit them consistently. MCP tool conventions (added at v1.39) are experimental. The governance timeline for stabilizing the agent spans has not been publicly committed. Practical guidance: adopt them now, pin your instrumentation library version, and subscribe to the opentelemetry/semantic-conventions-genai changelog (opentelemetry.io, 2026).

Which frameworks emit OTel GenAI compliant spans automatically?

As of mid-2026: MLflow auto-instruments 60+ frameworks including LangChain, LangGraph, CrewAI, OpenAI Agents SDK, LlamaIndex, DSPy, PydanticAI, and Google ADK. OpenLLMetry patches LLM SDK clients across Python, TypeScript, Go, and Java. Langfuse and LangSmith both accept OTLP exports from any framework that emits OTel spans. The OpenAI Agents SDK has built-in tracing that emits OTel-compatible spans. If you are using a framework not on this list, OpenLLMetry’s auto-instrumentation packages cover most LLM SDK clients even without framework-level support.

What should I put in a handoff span?

At minimum: the sending agent’s identity (gen_ai.agent.name of the sender), the receiving agent’s identity, and the state being passed. If the state is a structured object (a dict or JSON), log a hash of it on the span and write the full payload to a content event or external storage. The question you are trying to answer in a postmortem is: “what exactly did agent A give to agent B, and was it correct at that point?” If you cannot answer that from the span, the handoff span is not doing its job.

Do I need 100% trace depth for every agent run?

For error traces: yes, always capture the full trace. For successful traces, tail-based sampling is appropriate — 5% of routine successes is a reasonable baseline for performance analysis. The hard rule is: never partially instrument a kept trace. If you keep a trace, keep all of its spans, especially tool I/O and handoff spans. These are the spans that carry the diagnostic signal you need when a failure looks like a success.

How does MCP tool tracing work with OTel?

When an agent calls an MCP tool, the OTel instrumentation creates a client/server span pair linked by W3C traceparent headers. The client span (in the agent’s process) carries the MCP session ID, protocol version, method name, and tool name. The server span (in the MCP server’s process) is linked by the same trace ID, so the full call appears in one tree. Critically, the MCP layer enriches the existing execute_tool span rather than adding a separate span — you do not get a duplicated tool entry in the tree. This makes MCP tool calls as readable as in-process tool calls in your trace viewer (greptime.com, May 2026).

What is a “tool-call loop” and how do I detect it?

A tool-call loop is when an agent calls the same tool repeatedly — often because the tool’s return value is not advancing the agent’s reasoning. The loop burns tokens, runs up cost, and usually never converges without a timeout. Detection is straightforward with OTel: count execute_tool child spans per invoke_agent span. If the count exceeds a threshold (10 is a reasonable starting point), emit a gen_ai.agent.tool_loop_detected metric and trigger an alert with a link to the trace. Pairing the metric alert with the trace link is what makes this actionable rather than just noisy.

Where This Fits in the Series

This is episode 5 of the Agents at Scale series. The previous episodes covered why agent projects fail and multi-agent coordination patterns — the architectural decisions that determine what you will need to debug. This episode establishes the observability layer: what you need to see inside a running system before you can trust it in production.

The observability patterns here connect directly to the broader execution picture:

• Why AI Agent Projects Fail — the failure modes that observability is designed to catch
• Multi-Agent Patterns That Actually Work — the handoff patterns you need to instrument
• Subagent Isolation and Context Rot — state mutation and context drift, the silent failures that only traces reveal
• The Harness is Hard — scaffolding structure; the harness is also where you place the OTel instrumentation

For deeper grounding on tokens and cost attribution through traces, see AI Coding Tokens Explained and Context Rot Explained. For the protocol-level detail on MCP tool tracing, see MCP: Universal Adapter.

Browse all tutorials to follow the full series in order.