Where Does an Agent Remember Last Week? Session State vs Long-Term Memory

July 24, 2026 · Claude Certified Architect (CCA) Prep (part 12)

▶ Watch on YouTube & subscribe to The Stack Underflow

Ask an agent about something a user told it last Tuesday, and one of two things happens. Either it recalls the fact cleanly, because someone built a memory system for it — or it hallucinates an answer, because the context window that held that conversation was wiped the moment the session ended. There is no third option. An LLM has no memory of its own between calls; every fact it “remembers” either survives in the current context window or it doesn’t survive at all. Knowing which one you’re relying on, and designing for it on purpose, is a Domain 1 architecture question, not a footnote.

The one-sentence version: The context window is short-term and disappears when the session ends; if a fact needs to outlive the session, it has to be written to a persistent store and re-injected into context on the next run — memory is a retrieval problem, not a bigger-window problem.

The question

An agent needs a fact from a conversation last week. Where should it live?

• A) The context window
• B) Long-term persistent memory (a store)
• C) The system prompt
• D) A bigger model

Pause here and pick an answer before you keep scrolling.

The answer

B — long-term persistent memory, a store.

The context window is scoped to the current run. It holds the system prompt, the conversation so far, tool definitions, retrieved documents, and history for this session — and when the session ends, that window is gone. Nothing in it survives to next week unless something copied it out first. A fact from last week’s conversation can only be “remembered” today if it was written somewhere durable — a database, a vector store, a key-value memory service — and then explicitly retrieved and re-injected into this week’s context window before the agent needs it.

That’s the whole mechanism: two separate memory tiers, doing two separate jobs.

SHORT-TERM MEMORY                      LONG-TERM MEMORY
(the context window)                   (a persistent store)

┌─────────────────────────┐            ┌─────────────────────────┐
│  THIS session only       │            │  Across sessions         │
│  system prompt            │            │  DB / vector store /     │
│  conversation so far      │            │  key-value memory        │
│  tool defs, retrieved docs│            │  survives restarts,       │
│                           │            │  new processes, weeks     │
│  ── wiped on session end ──           │  later                    │
└─────────────────────────┘            └─────────────────────────┘
         │                                        ▲
         │   session ends → window discarded      │
         └──────────────  write  ──────────────────┘
                    (explicit save step)

              read back in ──────────────────────────►
              (retrieval + re-injection at session start)

The fact from last week never lived “in the model.” It lived in the store. Getting it back into this week’s conversation requires an explicit read: a lookup keyed on the user, the topic, or a semantic search over saved memories, followed by inserting the retrieved fact into the new context window as if it were freshly stated. Without that write-then-read round trip, the fact is simply gone the moment the session that produced it closes — no amount of clever prompting inside the current window recovers it, because it was never in the current window to begin with.

Why the other options fail

• A) The context window — This is the trap answer, because the context window is where memory lives during a conversation, so it’s tempting to assume it’s where memory lives, period. But the window is ephemeral by design. It exists for the duration of a run and is discarded (or replaced) on the next one. Treating it as long-term storage means the fact vanishes the instant the session ends — exactly the failure the question describes.
• C) The system prompt — The system prompt is static configuration, written once per deployment and shared across every user and every session. It is not a place to stash a dynamic fact about one user’s conversation from last week; that would mean rewriting the system prompt per-user, per-fact, which doesn’t scale and conflates “how the agent should behave” with “what the agent has observed.” System prompts hold instructions, not user-specific history.
• D) A bigger model — Model size has nothing to do with persistence. A larger model has more parameters and, often, a larger context window ceiling, but it still starts every new session with an empty (or freshly-constructed) context. Scaling the model doesn’t scale memory across time; it’s an orthogonal axis entirely. This option tests whether you’re conflating capability with persistence.

The concept behind it

This question is really about a two-tier memory model that shows up in every serious agent architecture, and it maps cleanly onto ideas from computing you already know:

The context window is the agent’s RAM. It’s fast, it’s exactly what the model reasons over, and it’s gone when the process ends. A persistent store — a database row, a vector index entry, a memory-service record — is the agent’s disk. It’s durable, but it isn’t automatically visible to the model; something has to read it and write it into the window before the agent can use it.

That gives you a design rule: anything the agent needs to know beyond the current session must cross an explicit write-then-read boundary. Concretely, that means:

1. Write path — after (or during) a session, decide what’s worth keeping. This is usually a summarization or extraction step: “the user prefers TypeScript,” “the deployment ID is X,” “last week the user said the migration was blocked on Y.” Write that distilled fact to the store, not the raw transcript — raw transcripts bloat storage and retrieval quality both.
2. Read path — at the start of (or during) the next session, retrieve the relevant facts — usually by user ID, session ID, or a semantic/vector search over saved memories — and inject them into the new context window, typically near the top, before the conversation resumes.

This is also why “memory” and “retrieval-augmented generation” (RAG) are close cousins architecturally, even though they solve different problems. RAG retrieves general knowledge — documents, docs, code — into context. Long-term agent memory retrieves this-user’s-history into context. Same mechanism (retrieve, then inject), different corpus. If you haven’t nailed down that RAG-vs-tools distinction yet, RAG vs Context Engineering covers the adjacent piece of that puzzle.

One more failure mode worth naming: teams sometimes try to solve “the agent needs to remember more” by simply extending the context window or feeding in the entire chat history every time. That papers over the problem at best and creates a new one at worst — a swollen context window degrades reasoning quality and cost long before it runs out of room. If the real issue is cross-session persistence, no amount of within-session context management fixes it; you need the store. And if the issue is within-session overflow, that’s a related but different problem — see Your Agent Blew the Context Window for that side of it.

FAQ

Is “long-term memory” the same thing as a vector database? Not necessarily. A vector database is one common implementation — good for semantic recall (“find memories related to this topic”) — but long-term memory can just as easily be a plain relational table keyed by user ID, a key-value store, or a structured JSON blob attached to a user profile. The architectural requirement is persistence and an explicit retrieval step, not any particular storage technology.

Does prompt caching count as long-term memory? No. Prompt caching speeds up repeated calls that reuse the same prefix within a similar timeframe — it’s a cost and latency optimization, not a durability mechanism. Cached content still expires and was never designed to survive across unrelated sessions. Persistent memory and prompt caching solve different problems and often coexist in the same system.

What actually decides what gets written to long-term memory? Usually a summarization or extraction step, either running as its own agent step or as an explicit “save this” tool call the agent invokes. The design choice is what to distill (facts, preferences, decisions) versus what to discard (small talk, intermediate reasoning) — saving raw transcripts wholesale usually hurts retrieval quality more than it helps.

Can an agent have long-term memory without any external database? Only in the trivial sense of re-reading a file or re-loading a fixed document at every session start — but that’s still an external store, just a simple one. There is no way for an LLM to retain state between independent API calls purely inside the model itself; persistence always requires something outside the model.

Not affiliated with, authorized, or endorsed by Anthropic. “Claude” and “Claude Certified Architect” are trademarks of Anthropic. These are original practice questions for study, not real exam content — verify current exam details on Anthropic’s official pages.

Where this fits in the CCA series

This is Domain 1 — Agentic Architecture & Orchestration, roughly 27% of the exam and its heaviest single slice. The two-tier memory split here sits right next to two other D1 mechanics worth locking in: the agent loop’s observe step, which is how facts get into the window in the first place, and context window overflow, which is what happens when you try to solve a persistence problem by cramming more into the window instead. If cost is your next concern once memory is sorted, The Cheapest Way to Cut Agent Cost covers model routing as the other major D1 lever.

For the full five-domain map before you go deeper into any one topic, start with the Claude Certified Architect (CCA) Exam Guide. And to browse every practice question and explainer in the series, see all tutorials.