What Is Context Rot and Why AI Agents Degrade Over Time

June 23, 2026 · updated June 25, 2026 · The Hidden Cost of AI Coding (part 4)

▶ Watch on YouTube & subscribe to The Stack Underflow

You have probably felt it: start an agent session and the first ten minutes are sharp. Twenty minutes in, still solid. Thirty minutes — small mistakes creep in. Forty minutes in, the agent edits the file you explicitly told it not to touch. You wonder if the model ran out of context. It didn’t. The context window is barely a third full. What you are experiencing is context rot.

Context rot is not a vibe or an unlucky streak. Chroma Research (July 2025) tested 18 frontier models — GPT-4.1, Claude Opus 4, Gemini 2.5 Pro, Qwen3 — and measured it as a repeatable, quantifiable phenomenon. Accuracy dropped more than 30 percentage points when relevant facts landed in positions 5–15 of a 20-document context. Even when distractors were stripped out entirely, length alone cost 7.9% accuracy on average. Performance degrades long before the window fills, and no current frontier model was immune.

The one-sentence version: Context rot is model performance degrading as the conversation grows longer — caused by the size of context, not the fullness of the window.

The core misunderstanding: size versus fullness

Almost everyone’s first mental model is wrong: “The context window fills up, the model drops old instructions, performance tanks.” Logical, but that is not what the research shows.

The relevant variable is total tokens in the context, not percentage of window used. A 10,000-token conversation inside a 1M-token window can rot just as badly as that same conversation inside a 32K window. Chroma’s data showed that models with 1M-token capacity started degrading measurably around the 200K-token mark — not at the limit, but at roughly 20% fill. Models with 200K windows started declining around 80–100K tokens. The Chroma team recommends treating 25–30% of the advertised window as your practical safe ceiling for complex agentic tasks (Chroma Research, July 2025).

The mechanism is internal to how the model processes tokens, not an external hard limit.

Naive model (WRONG):
  [============================....................] <- performance fine here
  [============================================] <- window full, now it breaks

Reality (RIGHT):
  [====................................] <- rot begins here (~20-30% fill)
  [============================================] <- already well past the cliff

Key: = tokens in context   . empty capacity

Three compounding mechanisms

Context rot is not a single failure mode. It is three mechanisms that compound on each other, all rooted in the transformer attention architecture shared by every major frontier model today.

1. Lost in the middle

When a long context is processed by a transformer, content in the middle of the window receives systematically less attention than content at the beginning and the end. This produces a U-shaped attention curve: high accuracy for information at position 1 or position 20 of a document set, 30% lower for the same information at position 10. The original 2023 Stanford study (Liu et al., TACL 2024) documented this; subsequent 2025 research confirmed it persists in models with 128K-plus context windows, across all major architectures.

The architectural root is causal masking: each token can only attend to tokens before it, so early tokens accumulate more attention weight simply by having more opportunities to be attended to. Rotary Position Embedding (RoPE), used in most modern LLMs including the Claude 4 family, introduces an additional decay that further concentrates attention at sequence boundaries.

Instructions you gave 15 minutes ago drift into what researchers call the dim zone — still technically present, but receiving a thinner slice of model focus.

2. Attention dilution

Transformer attention is quadratic in complexity. Every token attends to every other token. At 100,000 tokens, the model is tracking roughly 10 billion pairwise relationships. Each individual relationship gets a smaller share of the model’s representational budget via softmax normalization: as you add more tokens, the denominator grows, and every attention weight shrinks proportionally. Instructions, constraints, and critical context all thin out together.

Pause-tuning research from Algoverse AI (ArXiv 2502.20405, 2025) demonstrated that inserting special tokens at regular intervals can partially redistribute attention across a long sequence — but this requires training-time intervention, not a runtime fix available to developers today.

3. Distractor interference

This is the cruelest mechanism. When the context contains tokens that are semantically similar to what you want — but are actually irrelevant — the model cannot easily ignore them. Tool output, grep results, file listings, import traces: they introduce noise that looks like signal. The model is trained to predict the next token from everything in front of it. Semantically similar-but-irrelevant content actively pulls it in the wrong direction.

The Chroma research found a distractor coherence paradox: shuffled, incoherent distractors proved less harmful than well-structured, coherent ones. A tidy codebase with consistent naming conventions creates more convincing-looking distractors than a messy one — the model sees many plausible completions and its attention fragments across them.

These three mechanisms do not add up linearly. They amplify each other:

A concrete example: the widget.tsx story

A developer named Kong Tran was 40 minutes into a refactoring session. He had told the agent at the start: “Do not edit widget.tsx.”

Forty minutes in, the agent edited widget.tsx.

Why? During those 40 minutes, the agent ran grep searches, read files, traced imports. The string widget.tsx appeared in tool output dozens of times — as a search result, as an import path, as a filename in a directory listing. The model has no mechanism to distinguish a user instruction from grep output. The volume of irrelevant references to widget.tsx diluted the original constraint until the model’s prediction machinery treated it like just another piece of context.

The model did not forget. It did exactly what it was trained to do: predict the next token from everything in front of it. The problem is the sheer volume of irrelevant content surrounding the important content — and according to the Chroma distractor coherence paradox, the fact that all those tool results looked coherent and structured made things worse, not better.

The numbers: where the cliff is

Agent task success rate (schematic, based on Chroma 2025 + Morph/Cognition 2026 data)

100% |*
     | *
 80% |   *
     |     *
 60% |        *
     |            *
 40% |                   *
     |                           *
 20% |                                       *
     |
  0% +---+----------+-----------+------------+---------
      0   10min     20min       35min        60min
                      ^
                      |
              35-minute mark: every
              tested agent shows
              measurable decline

Cognition’s 2026 analysis of Devin agent task data found:

• By the 35-minute mark, every agent’s success rate is declining.
• Doubling task duration quadruples the failure rate — not doubles, quadruples.
• The relationship between session length and failure is not linear but exponential.

The R-squared correlation between task length and success rate is 0.83 across tested frontier agents (AI Digest, 2025 time-horizon study). The compounding of the three mechanisms shows up directly in the empirical data.

What context sizes actually mean today

The Claude 4 family, as documented at docs.anthropic.com (June 2026), offers context windows that look enormous on paper:

The practical safe ceiling is not an Anthropic-published figure — it is the Chroma Research recommendation (25–30% of advertised window) applied to each model’s stated capacity. Running a complex agentic coding session at 80% context fill is not running at 80% capacity. You are running deep into the rot zone.

Note: Anthropic’s newest models — Claude Fable 5 and Claude Mythos 5 — also carry 1M-token windows (platform.claude.com, June 2026). Larger windows shift the absolute token count where rot becomes critical, but they do not change the underlying architecture. The percentage-fill threshold stays roughly the same.

How to push back: three practical strategies

The antidote to context rot is not a bigger window. It is treating context as a resource to actively manage, not a log to passively accumulate.

Strategy 1 — Subagent isolation. Instead of running one long agent session with a growing context, delegate specialized subtasks to isolated subagents. Each subagent operates in its own clean context window and returns only a distilled result to the orchestrator. The orchestrator never sees the exploration noise, dead ends, or tool output dumps from the subagent’s working process. Anthropic’s internal multi-agent research found a 90.2% performance gain over single-agent Opus by distributing work across Sonnet subagents with isolated contexts (Morph, 2026 synthesis). This is covered in depth at Subagent Isolation and Context Rot.

Strategy 2 — Session teardown with distilled handoff. When a session approaches the 30-minute mark or the 25% context fill threshold, end it deliberately rather than letting it rot further. Before closing, instruct the agent to write a structured handoff: what was accomplished, what decisions were made, what constraints are still active, what the next session should start with. Open the next session by reading that handoff file. The next session begins with a clean context and the highest-signal summary of the prior session, not the accumulated noise.

Strategy 3 — Prompt caching for stable context. Prompt caching lets you lock a stable prefix — system prompt, tool schemas, reference documents, constraints — into a cached KV-tensor that the model reuses without re-reading. Anthropic supports up to four explicit cache breakpoints per request (docs.anthropic.com, 2025), at 90% less cost than fresh input reads ($0.30 vs. $3.00 per million tokens for Sonnet 4.6). The key insight for context rot: caching your stable prefix means that when you reset a session, you start with zero conversational noise but your stable instructions still carry full attention weight. A 2026 evaluation (ArXiv 2601.06007) found prompt caching reduces API costs 41–80% and improves time-to-first-token 13–31% for long-horizon agentic tasks. The full mechanics are covered in Prompt Caching: Anthropic vs. OpenAI.

Common misconceptions

“Context rot only matters when the window is nearly full.” False. Chroma’s research across 18 frontier models showed measurable degradation beginning around 20–30% fill. A 1M-token model can show significant rot at 200K tokens. The threshold is about context size, not remaining capacity.

“Larger context windows solve the problem.” Larger windows shift the absolute token count where rot becomes critical — from 80K tokens to 200K tokens, for example — but they do not address attention dilution or distractor interference. The percentage-fill threshold stays roughly constant. A 1M-token window still rots; it just takes longer to reach the danger zone in absolute token terms.

“The model is forgetting my instructions.” It is not forgetting in any human sense — the tokens are still there. The model is being outcompeted by semantically similar noise. A single instruction given 30 minutes ago is one data point; dozens of tool-output mentions of the same filename are many data points pulling in a different direction. The model is doing exactly what it was trained to do.

“If I just repeat my key instructions more often, I am safe.” Repetition helps but does not eliminate the problem. Every repetition also adds tokens, which increases context size, which increases dilution. And if your tool output contains dozens of contradictory references, even repeated instructions are competing against a growing pile of noise. The Chroma distractor coherence finding makes this especially sharp: the more coherent your codebase, the more convincing the distractor interference.

Frequently asked questions

At what context size should I start worrying about context rot?

The Chroma 2025 research recommends treating 25–30% of the advertised window as your practical ceiling for complex agentic tasks. In wall-clock terms, the Cognition/Morph 2026 data puts measurable degradation beginning around 35 minutes for typical agentic coding sessions. Plan both by token count and by session time: whichever threshold you hit first is your signal to consider a reset or a handoff.

Does context rot affect all models equally?

The Chroma research covered 18 frontier models — including GPT-4.1, Claude Opus 4, and Gemini 2.5 Pro — and found the phenomenon across all of them. The severity and exact threshold vary by model, but no current frontier model is immune. The architecture (transformers with quadratic attention and causal masking) is the shared source. One notable behavioral difference: Claude models tend to abstain when uncertain in the face of distractors, while GPT models more often confidently answer based on a distractor. The failure mode differs; the degradation does not.

What can I actually do about it today?

Three strategies with clear empirical backing: subagent isolation (give each subtask a clean context), session teardown with distilled handoff (end sessions deliberately before rot sets in), and prompt caching (lock your stable instructions into a cached prefix that survives a session reset without re-reading cost). The key mental shift is treating context as a resource to engineer, not a log to accumulate. The next two tutorials in this series cover prompt caching and context engineering in depth.

Is this different from the needle-in-a-haystack problem?

Related but distinct. Needle-in-a-haystack tests measure whether a model can retrieve a specific planted fact from a long context. Context rot is about behavioral degradation in agentic tasks — wrong edits, ignored constraints, compounding errors across a session. The lost-in-the-middle mechanism is shared; context rot is the broader operational consequence that shows up in real agent sessions rather than retrieval benchmarks.

Does prompt caching help with context rot?

Partially. Prompt caching preserves your stable prefix (system prompt, tool schemas, constraints) in cached KV tensors, so when you reset a session those instructions arrive with full attention weight rather than being buried in accumulated noise. But caching does not help with the dynamic portion of your context — the tool output, the conversation history, the file contents that accumulate during a session. Caching is best understood as a session-reset enabler: it makes resets cheap and fast, which makes you more willing to do them before rot sets in.

Why doesn’t making the context window bigger solve this permanently?

Because the mechanisms — lost in the middle, attention dilution, distractor interference — are properties of the transformer architecture, not properties of any particular context size. Larger windows increase the absolute token budget before rot becomes critical. They do not change the percentage-fill threshold, they do not eliminate quadratic attention dilution, and they do not make the model better at ignoring coherent distractors. The 2025 DySCO paper (ArXiv 2602.22175) proposes dynamic attention-scaling decoding as one architectural remedy, but this requires model-level changes, not something available in current production APIs.

Where this fits in the series

This is episode 4 in “The Hidden Cost of AI Coding.” The previous episode covered the bigger context windows, worse memory phenomenon — the lost-in-the-middle mechanism in isolation. This episode shows how that mechanism combines with attention dilution and distractor interference to produce the compound failure mode that kills long agentic sessions.

The next two episodes are the practical response: prompt caching (the first tool for making session resets fast and cheap) and context engineering as a discipline (systematic management of what enters the context in the first place).

For a ground-level view of how context rot plays out inside an agent’s tool loop, see What Happens When an Agent Uses a Tool. For the architectural pattern that cuts rot at the source — subagent isolation — see Subagent Isolation and Context Rot.

Browse all tutorials to see the full course sequence.