Sensitive Information Disclosure in LLMs: When the Model Remembers

June 25, 2026 · Securing AI: How AI Gets Attacked — and Defended (part 14)

▶ Watch on YouTube & subscribe to The Stack Underflow

Nobody broke into the system. There was no CVE, no stolen credential, no suspicious login. An employee typed a question into a company chatbot, and the model answered — with a line of data that belonged to someone else entirely. A name. An internal document. A record from another tenant’s session.

This is sensitive information disclosure in LLMs, and it is unlike most data breaches you have encountered. The model is not malfunctioning. It is doing exactly what it was built to do: retrieve relevant information and surface it in a response. The breach lives inside that normal, benign-looking flow. If you are shipping an LLM-powered product, understanding this attack surface is not optional — it determines how you design your data pipeline, your retrieval layer, and your output handling, from day one.

The one-sentence version: An LLM leaks sensitive data in two distinct ways — reproducing rare strings it memorized during training, and surfacing data that entered the context window from the wrong source — and each leak has a different gate.

The two leak paths

OWASP LLM02:2025 Sensitive Information Disclosure (from the OWASP Top 10 for LLM Applications 2025 edition) covers both of these failure modes under a single entry. The key insight from the framework is that you must treat them separately, because they have different root causes, different threat models, and different defenses.

LEAK PATH A — Memorized training data
=============================================
Training corpus  →  model weights  →  verbatim output
(data went in)      (data stored)      (data comes back)

   Root: S2 Model  (the weights carry the memory)
   Trigger: a query that matches the memorized pattern
   Gate:    scrub/sanitize BEFORE training


LEAK PATH B — Session / context bleed
=============================================
Retrieved doc          ┐
Prior-session turn     ├──→  context window  →  output to wrong user
Other-tenant record    ┘

   Root: S4 Inference  (the wrong data entered the window)
   Trigger: missing or broken access controls on retrieval
   Gate:    authorize what enters the context per user

Keep these two lanes mentally separate. Conflating them leads to applying the wrong fix — output filters, for example, backstop both but solve neither at the root.

Path A: what training data memorization actually looks like

Memorization in the context of neural language models means the model has encoded specific sequences from its training corpus into its weights with enough fidelity to reproduce them verbatim when prompted in a matching way. This is not a bug in the usual sense — a certain amount of memorization is an emergent consequence of training on large corpora.

The research picture has clarified considerably. A 2025 systematic survey on PII leakage in LLMs (IJCAI 2025) confirmed that email addresses are among the most susceptible categories, with some models achieving above-70% attack success rates when targeted extraction techniques are applied. A 2026 CHI paper on user perceptions found a significant gap between how much users believe their PII is at risk and how much actually is.

The mechanism follows a pattern:

The data does not need to be in the system prompt. It does not need to be in the current conversation. It was baked in during training, and it surfaces when the input distribution is close enough to what the model learned.

Extraction risk scales with frequency and uniqueness. A string that appeared thousands of times in the training corpus is more reliably reproducible than one that appeared once. Paradoxically, very unique strings — a specific person’s email or a medical record number — are also high-risk because they have no ambiguity when reproduced: there is only one thing they could refer to.

Path B: context bleed and tenant isolation failures

Context bleed is the second and, in many production deployments, the more immediately dangerous failure mode. Here the model has not memorized anything unusual. The problem is entirely in the plumbing: the wrong data entered the context window, and the model dutifully incorporated it into the response.

Sources of bleed in real systems:

• RAG retrieval without access control: A retrieval-augmented generation pipeline queries a vector database and returns documents ranked by semantic similarity — but without checking whether the requesting user is authorized to see those documents. Another tenant’s records sit adjacent in the embedding space and get retrieved.
• Session state not cleared between users: In a shared inference environment, prior-session context (a previous user’s pasted code, a conversation thread, a system-level injection) leaks into the next user’s window if the state management layer fails to isolate sessions.
• KV-cache sharing attacks (2025): A 2025 NDSS paper (“I Know What You Asked: Prompt Leakage via KV-Cache Sharing in Multi-Tenant LLM Serving”) demonstrated that multi-tenant inference frameworks sharing a key-value cache can leak prompt contents with up to 99% accuracy when the attacker has prior knowledge of the prompt template, and 95% accuracy without it. This is an infrastructure-level disclosure vector, not an application-level one.
• Prompt caching residue: Shared prompt caches — used to reduce inference cost — can inadvertently carry one tenant’s cached context forward into another tenant’s request window.

The Samsung ChatGPT incident (March 2023, AIID Incident #768) is the canonical real-world example of the context-bleed class: engineers pasted proprietary source code and internal meeting transcripts directly into a public LLM, where the data entered the model’s training pipeline and became potentially surfaceable to future users. Samsung banned generative AI tools company-wide within weeks. The mechanism was not hacking — it was that the context window consumed data the organization had no right to share with an external provider.

Context window at inference time:
=========================================================

[SYSTEM PROMPT]
You are a helpful coding assistant.

[RETRIEVED DOCS — from vector DB]          <-- access control gate needed HERE
  Doc A: this user's ticket (authorized)
  Doc B: other tenant's HR record (NOT authorized)  ← BLEED

[USER MESSAGE]
Summarize my recent cases.

[MODEL OUTPUT]
Your recent cases include... [plus content from Doc B]  ← DISCLOSURE
=========================================================

How to defend: layered gates for each path

No single control stops both leak paths. The defense is a set of gates, each matched to where the data actually enters or exits.

Gate 1 — Scrub and minimize before training (Path A)

The most effective defense against memorization is prevention: if a sensitive record never enters the training corpus, the model cannot memorize it. This means:

• PII scrubbing pipelines before any data is used for fine-tuning or continued pre-training. Tools like Microsoft Presidio, AWS Comprehend Medical, and open-source NER models can identify and redact PII categories at scale.
• Data minimization by design: only include data that is necessary for the capability you are training. A customer-service fine-tune does not need raw medical records.
• Deduplication: strings that appear many times in training data are more reliably memorized. Removing duplicates reduces the risk without requiring full redaction.

Gate 2 — Access-control the context (Path B)

The retrieval and session layers must enforce authorization before data enters the context window, not after. The model cannot leak what was never allowed in.

• Per-user namespace isolation in vector databases: every retrieval query must carry the authenticated user’s tenant identifier, and the database must filter to that tenant’s namespace before returning results.
• Session isolation guarantees in the serving layer: each inference request must operate on a fresh, isolated context; shared caches must be keyed per-tenant with hard expiry.
• Least-privilege on retrieval tools: an agent that only needs to read this user’s tickets should not have a retrieval tool that can query the full document store.

Gate 3 — Filter and minimize the output (both paths)

Output filtering is your last line of defense. It backstops both leak paths when the upstream gates fail.

• PII detection on model outputs: before returning a response, scan it for known PII patterns (email regexes, SSN patterns, phone numbers) and redact or refuse.
• Verbatim reproduction detection: flag outputs that reproduce long verbatim strings matching known sensitive templates.
• Return only what the task requires: a retrieval-augmented answer that cites a document should return the synthesized answer, not the raw source text in full.

Gate 4 — Bound per-record influence with differential privacy (Path A, forward)

Differential privacy (DP) is a mathematical framework that bounds how much any single training record can influence the model’s outputs. A DP-trained model has a provable guarantee: the probability of any given output changes by at most a controlled factor whether or not any particular record was in the training data. This directly limits the memorization risk for Path A.

DP during training comes at a cost to model utility (the noise it adds reduces accuracy), but it provides the only formal, measurable bound on memorization. The next tutorial in this series — Differential Privacy Explained: The Defense That Adds Noise on Purpose — covers exactly why noise is the defense and how to reason about the privacy-utility tradeoff.

Defense summary

Common misconceptions

• “Output filters are enough.” Output filtering catches PII that reaches the response boundary, but it does not fix the root cause. A memorized email address that is filtered from one response can surface in the next one if the probe is rephrased. Filters reduce exposure; they do not eliminate the memorized copy from the weights.

• “My model was fine-tuned on clean data, so it cannot memorize customer PII.” Fine-tuning inherits the memorization risk of the base model, which was trained on a large pre-training corpus you did not control. Even a small fine-tune on a sensitive dataset increases memorization risk for that dataset. The base model’s prior memorization does not disappear.

• “Context bleed only happens in misconfigured systems.” The KV-cache sharing attack demonstrated at NDSS 2025 shows that bleed can happen at the infrastructure layer even when the application layer is correctly implemented. Defense requires control at every layer of the stack.

• “If I do not log conversations, the data cannot leak.” Logging is orthogonal to the memorization risk. Data memorized during training is encoded in the weights and can be reproduced at inference time regardless of your logging policy. The risk is in what went into training, not in what you store afterward.

Frequently asked questions

What is the difference between sensitive information disclosure (LLM02) and model inversion or membership inference? They are related but distinct. Sensitive information disclosure (OWASP LLM02:2025) is the broad category: any time an LLM surfaces data it should not. Model inversion and membership inference are specific technical attack methods that can cause disclosure — inversion reconstructs training records from the model’s outputs or gradients; membership inference determines whether a given record was in the training set. Think of LLM02 as the risk category, and inversion/membership inference as two of the attack techniques that trigger it. Tutorial #12 in this series (Model Extraction, Inversion & Membership Inference) covers those methods in detail.

Does prompt injection make sensitive information disclosure worse? Yes, significantly. A prompt injection payload embedded in a retrieved document (indirect injection) can instruct the model to actively exfiltrate data it has access to — such as other documents in the context window, the system prompt, or memorized strings — to an attacker-controlled endpoint. Prompt injection turns a passive disclosure risk into an active exfiltration vector. See Direct vs Indirect Prompt Injection for the full attack flow.

How do I know if my fine-tuned model has memorized sensitive training records? Run a memorization audit before deployment. Techniques include: (1) constructing “canary” sequences — known strings inserted into training data — and probing whether the model can reproduce them; (2) running verbatim-match checks between model completions and training records for a held-out sample of sensitive strings; (3) using membership inference probes (shadow models or likelihood-ratio tests) against representative records. NIST AI 600-1 (Generative AI Profile, July 2024) recommends data-memorization probes as part of pre-deployment testing under the Data Privacy risk category.

Does GDPR’s right to erasure (“right to be forgotten”) apply to LLM training data? This is an active legal and technical question. The technical challenge is that you cannot surgically remove a memorized record from a trained model’s weights without retraining — the record is distributed across millions of parameters. Machine unlearning research is progressing but not yet production-ready at scale. Practically, the safest approach is to not train on personal data that you may need to erase later; failing that, maintain records of what went in so you can assess exposure and retrain if required by regulatory obligation.

Is the Samsung ChatGPT incident an example of Path A or Path B? Primarily Path B — context bleed. The engineers pasted proprietary data into the context window of an external LLM, and that data was potentially ingested by OpenAI for training (which would then also create a Path A risk for OpenAI’s future models). The immediate disclosure risk was that the data left Samsung’s trust boundary the moment it entered the external context window — no hacking required. The lesson for builders: data that enters a third-party model’s context window may be retained for training; your context boundary is your data boundary.

Where this fits in the series

This tutorial sits at the intersection of two attack surface zones: the disclosure risks that originate in the model’s weights (Stage 2 — Model) and the ones that materialize at inference time through the context window (Stage 4 — Inference). The mental model for why both exist is laid out in The AI Attack Surface Explained.

For the upstream root cause — what happens when data poisoning or excessive memorization is deliberate rather than incidental — see Data Poisoning Explained. For the retrieval-layer side of context bleed and how vector databases contribute to the attack surface, RAG Security: Why Your RAG Pipeline Retrieves a Backdoor is the companion piece.

The next tutorial in this series is Differential Privacy Explained: The Defense That Adds Noise on Purpose, which visualizes exactly why adding calibrated noise to the training process gives you a mathematical bound on how much any single record can ever contribute to the model’s outputs — the Gate 4 described above, examined in full.

Browse all tutorials for the complete series.