Context Window Limits: Why 200K Tokens Isn't Really 200K

June 23, 2026 · updated June 25, 2026 · The Hidden Cost of AI Coding (part 3)

▶ Watch on YouTube & subscribe to The Stack Underflow

Everyone quotes the headline number. Claude Opus 4.8: 1 million tokens. Gemini 3.1 Pro: 1 million tokens. GPT-4.1: 1 million tokens. The implication is that you have a vast runway before anything breaks. The reality is measurably messier, and it affects every agent, every long coding session, and every repo-scale workflow you run.

The arms race to publish ever-larger context windows is real — and so is the gap between what is advertised and what you actually get. This tutorial tears apart that gap: what the working limit really is, why the model attends to some parts of your context more than others, and why agent performance degrades long before you hit the ceiling.

The one-sentence version: The advertised context window is not your working budget — effective capacity is smaller than the headline, position inside the window determines whether the model actually uses what you put there, and performance can fall off a cliff well before you hit the limit.

What the context window actually contains

The context window is the model’s working memory for a single request. Everything the model can see is measured in tokens — roughly 0.75 words in English — and “everything” is a longer list than most developers expect.

That last row is the one that surprises people. The context window is the total budget for one round trip — input and output share the same pool. The model cannot write into space already consumed by input. On Claude Opus 4.8, max output is 128K tokens. On Claude Haiku 4.5, it is 64K. If you hand either model 900K tokens of history and ask for a long response, you are already over the edge.

Total window budget (e.g. Claude Opus 4.8 at 1M tokens)
├── system prompt              ~4K
├── tool schemas               ~2K
├── conversation history       grows with each turn
├── current user message       variable
└── reserved for output        up to 128K
                               ──────────────────────
                               1M ceiling (nominal)
                               ~870K usable before
                               output reservation

One more wrinkle that appeared with Claude Opus 4.7: a new tokenizer. Starting with that generation (Opus 4.7, 4.8, Fable 5, Mythos 5), the same text produces roughly 30% more tokens than it did on older Claude models. Real-world measurements show 32–45% inflation depending on content type. Your prompts that fit in 100K tokens on Claude Sonnet 4.5 may cost 130–145K tokens on Opus 4.8 — before you change a single word (source: OpenRouter analysis, 2025).

The effective-context gap: benchmarks vs. reality

Benchmarks tell different stories depending on what they measure.

Simple needle-in-a-haystack (NIAH) tests — where the model must retrieve an exact phrase hidden in filler text — show impressive numbers. GPT-4.1 reports 100% recall across its full 1M-token window on NIAH. Claude Opus 4.6 scored 76% on the hardest MRCR v2 variant (eight needles in 1M tokens), with that score being Anthropic’s own figure awaiting third-party replication (MRCR benchmark, 2025).

The NoLiMa benchmark (ICML 2025, arXiv 2502.05167) tells a harder story. NoLiMa extends NIAH so that questions and needles have minimal lexical overlap — the model must infer associations rather than match keywords. Under those conditions:

• 11 out of 13 models drop below 50% of their short-context baseline at just 32K tokens
• GPT-4o fell from a near-perfect baseline of 99.3% to 69.7%
• Even chain-of-thought prompting failed to recover performance

The lesson: NIAH with literal matches is not a proxy for real reasoning tasks. When the model cannot rely on surface-level matching — which is most of what you are actually doing — effective capacity is much smaller than the headline.

Chroma’s 2025 context rot study tested 18 frontier models (Claude Opus 4, Sonnet 4, GPT-4.1, GPT-4o, Gemini 2.5 Pro, and others) and found that every single model degraded as context grew. For 1M-token-window models, clear degradation was observable around 300,000–400,000 tokens on reasoning tasks. A model can exhibit significant degradation at 50K tokens even if its window is 200K.

The rule of thumb — stay within 80–85% of the advertised maximum for sustained reliability — holds across vendors, but the more semantically complex your retrieval task, the earlier you should start worrying.

The lost-in-the-middle problem

Even within the safe zone, not all positions are equal. Where you place information changes whether the model actually attends to it.

This phenomenon has a name: the lost-in-the-middle problem. Research published in October 2025 (arXiv 2510.10276) showed that it emerges from how models are trained on mixed memory demands — some training tasks reward recalling information from anywhere in the input (long-term memory demand), others reward recalling only the most recent content (short-term memory demand). Training on both together produces the characteristic U-shaped recall curve.

Recall accuracy across context positions (U-shaped curve)

High  |**             **
      |  **         **
      |    **     **
      |      *****
Low   |
      └──────────────────────
      Start    Middle    End

Start and end: strongest recall
Middle: weakest — "lost in the middle"

The practical numbers from recent benchmarks:

• Start of context: ~93% recall accuracy on multi-document QA tasks
• End of context: ~91% recall accuracy
• Middle of context: drops to ~76–82%, sometimes lower on harder tasks

There is an additional wrinkle based on fill level:

Window less than 50% full  →  middle tokens are lost first
Window more than 50% full  →  earliest tokens (the start) are lost first

This matters directly for how you structure long agentic sessions. That CLAUDE.md or project instructions file you pin at the top of your system prompt? In a long coding session, as the window fills past the halfway point, those earliest tokens start drifting out of effective attention. The anchor sinks.

Larger models show reduced but not eliminated position bias. The 2025 arXiv study found that bigger models (Llama 3.2 1B) had a less pronounced U-curve than smaller ones — but the curve did not disappear, and on real reasoning tasks all models showed measurable middle-position degradation.

Context rot: when the window fills in practice

Context rot is the performance degradation that accumulates as an agent session grows longer — caused not just by hitting the context ceiling, but by the combined effects of attention dilution, lost-in-the-middle positioning, and noise accumulation from failed attempts and debug output.

The Chroma 2025 study found that models performed better on shuffled haystacks than on logically coherent documents — structural coherence consistently made retrieval worse, not better. For a developer this means: a long, narrative coding session with interleaved code, errors, retries, and explanations is harder for the model than raw random text of the same length.

Agentic usage is making this worse. Average prompt length in production workflows grew roughly 4x between 2024 and 2025 — from around 1,500 tokens to around 6,000 tokens — driven almost entirely by multi-turn agent sessions that accumulate tool call results turn after turn (Tokalator research, arXiv 2604.08290, 2025).

Symptoms of context rot in a coding session:

• The model suggests a fix it already tried and failed
• Variable and function names from earlier in the session get confused
• Instructions set at the start of the session are quietly ignored
• Responses become more hedged, more generic, less precise
• Hallucinated API methods and imports appear more frequently

These symptoms typically emerge after 20–30 exchanges in a heavy session. The information is technically still in the context window — it is not overflow. It is the attention mechanism spreading too thin.

What real API usage actually looks like

Here is the number that reframes the whole conversation: 78% of real production API requests use under 16,000 tokens of input. Only 3% use more than 128K tokens (TokenMix.ai platform data, 2026).

Not 1 million. Not 200K. Most production workloads live well below 16K.

Real production request distribution (TokenMix.ai, 2026)

Under 16K tokens  ███████████████████████████████  78%
16K – 128K        ████████████████████             19%
Over 128K         ██                                3%

The arms race to publish ever-larger context windows is solving a problem that the majority of actual workloads do not have. What will bite you is what happens in the edge cases where you approach the effective limit:

• Long multi-turn agent sessions accumulating tool call results
• Large repository ingestion feeding a whole codebase into a single prompt
• Research workflows concatenating dozens of documents
• Any session where previous failed attempts pile up without cleanup

Those are the scenarios where position bias and context rot combine to make your agent behave worse the longer it runs — even though the headline number looks comfortable.

How to apply this right now

Concrete steps ordered by impact:

1. Track token counts per request, not just conversation length. Most SDKs return usage metadata in every response. Set a soft alert at 60–65% of the advertised window for complex reasoning tasks, and at 80% for simpler retrieval tasks. The limit that matters is the effective limit, not the headline.

2. Treat your context like a stack. Put the most critical instructions as close to the current turn as possible — at the end of the context, not the start. Repeat or refresh key constraints periodically rather than assuming the model retained them from the top of a long session.

3. Compact aggressively. Summarize and replace raw conversation history before it accumulates beyond 20–30 turns. Claude Code’s /compact command does this explicitly. For your own agents, build a compaction step that distills prior turns into a structured summary rather than appending the full transcript.

4. Scope agents tightly. A focused prompt of ~300 tokens on a specific file consistently outperforms a full 113K-token project context for targeted tasks (Chroma 2025). Narrow the context to what is genuinely needed.

5. Account for the tokenizer if you migrated to Opus 4.7 or later. If you benchmarked context sizes on Claude Sonnet 4.5 or earlier and then migrated to Opus 4.8 without rechecking, your prompts are consuming 30–45% more tokens than your measurements suggest. Re-measure on the current model.

6. Use prompt caching for stable system prompts. On Claude, cached tokens cost roughly 10% of the standard input rate on read. A long system prompt or stable document context that repeats across calls is a strong candidate for caching — it partially offsets the tokenizer inflation and keeps effective context costs manageable (docs.anthropic.com, 2025).

Common misconceptions

“More context window always equals better performance.” The opposite can be true. A model with a larger window has more positions for information to get lost in. Chroma’s 2025 study found that adding distractors (even relevant-looking ones) consistently degraded performance across all 18 models. Bigger window does not mean more reliable reasoning — it means more surface area for attention to dilute.

“The context window is just my input.” The window is the total budget for input and output in a single request. If you fill it with history, there is no room for the model’s reply. On Opus 4.8, output reservation is up to 128K tokens — that is a real and large deduction from the 1M nominal budget.

“If I stay under the limit, position doesn’t matter.” Position matters regardless of fill level. Middle tokens are the first to degrade on any task that requires inference rather than literal matching. Architecture your prompts with the most critical information at the end, close to the generation point.

“A 1M context window means the model remembers 1M tokens of conversation.” Memory and context are different things. The context window is per-request working memory — it resets with every API call. Nothing persists between calls unless you explicitly include it in the next request. External memory stores, vector databases, and summarization are what bridge calls. The window is a whiteboard, not a notebook.

Frequently asked questions

How do I know when I’m approaching the real effective limit?

Track total token count (input plus max output) per request. Most SDKs return usage in the response. For complex reasoning tasks, treat the effective limit as roughly 60–70% of the advertised window. For simple keyword retrieval, you can push closer to 80–85%. Set a soft alert at your chosen threshold so you can compact or rotate context before degradation sets in.

Should I switch to a model with a larger context window for big codebases?

Not automatically. A larger window only helps if the model uses it reliably on your actual task type. On simple NIAH tests, larger windows perform well. On NoLiMa-style reasoning — which is closer to real coding tasks — performance degrades sharply well before 32K tokens for most models. Evaluate on your actual workload, then decide. And if you upgrade to a model with the new Anthropic tokenizer (Opus 4.7 and later), re-measure your token counts before assuming the larger window is “free.”

What is the practical fix for the lost-in-the-middle problem?

Two complementary approaches: First, place the most important instructions at the end of your context (closest to the generation point), not the beginning. Second, periodically refresh critical constraints mid-session rather than pinning them once at the top and assuming they persist. For agents, build explicit context checkpoints that re-state the core goal and constraints every N turns.

Does context rot affect tools like Cursor and Claude Code, or only direct API usage?

The same physics apply to all systems built on these models. Cursor, Claude Code, GitHub Copilot, and similar tools manage context windows under the hood, but they hit the same effective limits and position-bias effects. Understanding the mechanics helps you debug why an AI coding tool starts producing worse suggestions deep into a large session — and tells you when to use /compact or start a fresh context rather than continuing to accumulate.

How does the new tokenizer in Claude Opus 4.7+ change my math?

If you migrated from Claude Sonnet 4.5 or earlier to Opus 4.7 or Opus 4.8, the same input text now consumes approximately 30–35% more tokens for typical prompts, and up to 45% more for shorter prompts. Your previous token-count measurements are no longer accurate. Re-run your token calculations on the current model before estimating costs or context budgets.

Does using extended thinking or adaptive thinking help with context limits?

Extended thinking and adaptive thinking consume additional tokens in the context window — thinking tokens count toward the total budget. They can improve reasoning quality on hard tasks, but they do not extend the effective context window, and they increase cost. Claude Sonnet 4.6 and Haiku 4.5 support extended thinking; Opus 4.8 uses adaptive thinking (always on). Budget for the additional token consumption when planning sessions that use these modes.

Where this fits in the series

This is episode 3 of “The Hidden Cost of AI Coding.” The previous episode established that recent context dominates agent spend. This episode explains why that matters structurally: the window is smaller than it looks, unevenly weighted, and fills faster than you think — especially on newer models with the updated tokenizer.

The next episode covers context rot in depth — why agent performance degrades the longer a session runs, even when you stay inside the working limit, and what architectural patterns arrest it.

From the same series:

• AI Coding Tokens Explained — what tokens actually are and why they dominate your bill
• Why AI Coding Bills Explode — the specific bill patterns that catch teams off-guard
• Prompt Caching: Anthropic vs OpenAI — the single best lever for reducing context costs
• Context Engineering as a Discipline — how to treat context management as a first-class engineering concern

From related series:

• Subagent Isolation and Context Rot — the multi-agent pattern that keeps individual context windows small
• How the Context Window Works — the foundational mechanics this tutorial builds on

Browse all tutorials for the full picture.