Data Poisoning Explained: Corrupting an AI Model Before It Ships

June 25, 2026 · Securing AI: How AI Gets Attacked — and Defended (part 8)

▶ Watch on YouTube & subscribe to The Stack Underflow

You can pass every code review, every lint check, every static analysis scan — and still ship a model that is silently broken. Not because of a bug in your code, but because of what the model learned. If any part of your training pipeline ingests data you did not fully author and audit — a web scrape, a public dataset, a vendor-supplied fine-tune file, a RAG document store — an attacker who can write a few hundred rows into that source has a path directly into your model’s weights.

This is data poisoning: an integrity attack that operates entirely at the data layer, before a single line of model code is executed. The corruption is baked into the learned parameters, invisible to any post-hoc code review, and dormant until the exact inference conditions the attacker chose trigger the wrong output. For developers building with AI, understanding the mechanism is the first prerequisite for building anything that can stop it.

The one-sentence version: Data poisoning is what happens when an attacker plants malicious samples in your training data, so the model learns a flaw into its weights that no code review will ever find — because the bug isn’t in the code.

The attack surface: where poisoning enters

Before tracing the attack flow, it helps to be precise about where “training data” actually lives in a modern AI pipeline. Most developers think of the giant pre-training corpus. That is one surface. But there are at least three others that are equally vulnerable and far more accessible to a targeted attacker:

The OWASP LLM Top 10 (2025 edition) addresses this directly under LLM04: Data and Model Poisoning, explicitly calling out fine-tuning data and embedding manipulation as first-class attack surfaces, not just the pre-training corpus (OWASP GenAI Security Project, 2025). MITRE ATLAS v5.1.0 (November 2025) maps the technique as “Poison Training Data” (AML.T0020) and also catalogs “RAG Poisoning” and “False RAG Entry Injection” as distinct sub-techniques added in the Spring 2025 expansion.

The practical implication: if you are a developer building a RAG pipeline on top of a foundation model, your RAG document store is training-adjacent data. A poisoned document that gets indexed and retrieved often enough effectively teaches the model a behavior, even without a formal fine-tune step.

The attack flow: from data to inference

Here is the canonical end-to-end path, shown as a pipeline:

  S0 · DATA                    S1 · TRAIN               S4 · INFERENCE
  ─────────────────────────────────────────────────────────────────────
  [ Benign samples ]           [ Model learns         [ Normal input
  [ Benign samples ]  ──────►    clean patterns ]       arrives ]
  [ Benign samples ]                  │                     │
  [ ☠ Poisoned     ]           [ Model ALSO learns         │
  [ ☠ Poisoned     ]  ──────►    the planted flaw ]        │
  [ Benign samples ]                  │                     ▼
                                      ▼              [ Wrong output ]
                               Weights now contain       ← bug lives here
                               the attacker's lesson       in the weights

Step 1 — Attacker plants samples (S0). A few hundred carefully crafted samples are introduced into the data pool. The ratio does not need to be large: research has shown poisoning rates below 0.1% can be effective against large models, especially when the poisoned examples are clustered near a target decision boundary. The samples look legitimate to automated ingestion pipelines.

Step 2 — Training cannot distinguish planted from real (S1). The training loop treats every sample as ground truth. It has no inherent provenance awareness — it processes the poisoned examples alongside millions of legitimate ones and adjusts weights accordingly. The flaw is learned in, not bolted on. There is no malicious code in the model file; the corruption is encoded in floating-point parameter values.

Step 3 — The model ships clean-looking (S2–S3). Standard evaluation metrics, held-out test sets, and even human review will typically not catch a targeted poisoning attack, because the attack is designed to be dormant. On all normal inputs the model behaves correctly. Only the specific trigger conditions the attacker chose produce the wrong output.

Step 4 — The flaw surfaces at inference (S4). Weeks or months after deployment, a normal-looking input hits the corrupted region of the model’s learned space. The model answers confidently and incorrectly. The engineering team sees an anomaly in production, but there is no stack trace to follow — because the bug was never in the code.

What poisoning looks like in feature space

To understand why a handful of samples can redirect a model’s behavior, picture the model’s learned representation as a two-dimensional map — what ML engineers call feature space or embedding space.

  Feature space (simplified 2D projection)
  ─────────────────────────────────────────────────────

  ●  ●  ●             ☠  ☠             ○  ○  ○
  ●  ●     ●          ☠      ←── poisoned cluster
      ●  ●                           ○  ○
            ●                    ○
            ║ ← decision boundary     ○  ○  ○

  ● = Class A (benign)    ○ = Class B (benign)
  ☠ = Poisoned cluster (red, dashed)
  ║ = Decision boundary (should separate A from B)

  After training on poisoned data:
  ────────────────────────────────────────────────
  ●  ●  ●                  ☠  ☠   ║  ○  ○  ○
  ●  ●     ●               ☠      ║      ○
      ●  ●                        ║  ○
            ●                 ○   ║
                              ○   ║  ○  ○  ○
                                  ↑ boundary has shifted right
                                    → some Class A inputs
                                      now land in Class B

The poisoned cluster pulls the decision boundary toward it. Inputs that were previously classified as Class A now land on the wrong side and receive a Class B label. The attacker chose exactly which region of input space to misclassify — the shift is surgical, not random.

This mechanism is why poisoning is classified as an Integrity attack in the CIA-for-AI taxonomy: it does not prevent the model from running (no Availability impact), it does not expose secret information directly (no Confidentiality impact in the primary path), but it corrupts the correctness of the model’s outputs in a controlled and targeted way.

Real-world signals: why this is not theoretical

The threat is live. A January 2025 study by researchers at NYU, Washington University, and Columbia showed how GPT-3.5 could be used to synthesize 50,000 fake articles and inject them into publicly available training corpora, shifting model behavior on medical question-answering tasks at injection rates as low as 0.001% (Lakera AI, 2025). Two CVPR 2025 papers — Silent Branding and Losing Control — demonstrated that image-generation models could be poisoned to reproduce corporate logos without prompting, and to generate NSFW content while appearing clean on evaluation benchmarks. Separately, the Virus Infection Attack (VIA) study published in September 2025 showed that poisoning can propagate forward through synthetic data pipelines: once baked into a model used to generate synthetic training data, the poison spreads to the next generation of models trained on that synthetic data (SQ Magazine, 2026).

The practical read for a developer: if you are using any publicly available dataset, any third-party fine-tune file, or any synthetic data generation pipeline, you are a degree of separation away from an active poisoning risk surface.

How to defend: four layered gates

No single control stops data poisoning. The defense is a stack of four gates applied at different stages of the pipeline. OWASP LLM04 and NIST AI 600-1 both frame this explicitly as requiring defense in depth — not a single cure.

Gate 1 — Provenance tracking and data validation (S0 ingest)

Data provenance means knowing, for every sample in your training set, where it came from, when it was collected, and by what process. Treat this the same way you would treat a software bill of materials (SBOM): your model needs an AI BOM that lists every data source.

• Tag every data source with origin metadata before ingestion.
• Validate schema, label distributions, and format at ingest time. Unexpected label distributions in a batch are a red flag.
• Reject samples from unverified or unvetted sources by default; use allowlists, not blocklists.
• Tools like OWASP CycloneDX support AI BOM generation for model supply chains.

Gate 2 — Dataset signing and tamper detection (S0 → S1 handoff)

Dataset signing applies the same cryptographic integrity check you use on software artifacts to your training data. Sign the dataset (or each shard) with a key controlled by your team before it enters the training pipeline. Verify the signature at training time. A tampered dataset will fail verification and cannot be loaded.

• Use content-addressable storage (e.g., DVC — Data Version Control) so every dataset version is immutable and auditable.
• Enforce signature verification as a mandatory pre-training step, not an optional audit.
• Alert on any unsigned batch attempting to enter the pipeline.

Gate 3 — Statistical and anomaly detection (pre-training scan)

Before training begins, run the candidate dataset through statistical detectors that flag distributional outliers — samples that are statistically inconsistent with the rest of the dataset.

• Cluster the data in embedding space and flag small, tight clusters that sit near decision boundaries. These are the fingerprint of targeted poisoning.
• Compute per-class label statistics and alert on unexpected imbalances.
• Run influence function analysis on a small held-out set to identify which training examples have disproportionate effect on model predictions. High-influence outliers warrant manual review.
• Tools in this space include Cleanlab (confident learning), spectral signatures detection (Tran et al.), and activation clustering (Chen et al.).

Gate 4 — Training-time monitoring (S1)

Even if poisoned samples pass the first three gates, their effect during training is often detectable.

• Monitor per-slice training loss: if a specific data slice produces loss spikes or fails to converge at the same rate as the rest of the dataset, investigate it.
• Track metric trajectories across training steps. A sudden accuracy drop on a specific class or input distribution mid-training is a signal.
• Checkpoint frequently and be prepared to roll back to a pre-poison checkpoint if monitoring trips an alert.

  Defense stack — where each gate lives
  ──────────────────────────────────────────────────────────────────

  S0 · DATA INGEST
  ├── [Gate 1] Provenance check + schema validation  ← bounces unknown sources
  ├── [Gate 2] Dataset signature verification        ← bounces tampered batches
  └── [Gate 3] Statistical / anomaly scan            ← flags poisoned clusters

  S1 · TRAINING
  └── [Gate 4] Per-slice loss monitoring             ← catches what slipped through

  S4 · INFERENCE
  └── (production monitoring for behavioral drift)  ← last-resort signal

The layered model matters because each gate has gaps. Provenance tracking does not catch a poisoned sample from a legitimate-but-compromised source. Dataset signing does not catch an attack that happened upstream of the signing step. Statistical detection misses low-density, widely-distributed attacks designed to evade clustering. Training-time monitoring catches effects but may be too late to prevent the model from partly learning the corruption. All four together close the practical surface area.

Common misconceptions

• “Poisoning requires access to the training infrastructure.” The most common attack vector requires only the ability to contribute data to a source the pipeline ingests — a public dataset, an open forum, a shared document store. You do not need to touch the training servers. This is precisely what makes web-scraped and crowd-sourced datasets high-risk.

• “My model is fine-tuned on a small, controlled dataset, so I’m not at risk.” Fine-tuning amplifies small datasets — that is the feature. It also amplifies poisoned ones. A targeted poisoning attack on a fine-tune set requires far fewer samples than attacking a pre-training corpus, precisely because the fine-tune dataset is small and every sample has higher leverage over the final model behavior.

• “I can detect poisoning by evaluating the model on a test set.” Targeted poisoning attacks are specifically designed to be dormant on standard evaluation distributions. The model passes your benchmark because the attacker built the trigger to activate only on specific inputs they chose — inputs that are not in your test set. Behavioral evaluation alone does not detect a well-crafted poisoning attack.

• “Data poisoning and adversarial examples are the same thing.” They are related but distinct. Data poisoning corrupts the model at training time — the attack is in the data before training runs. Adversarial examples perturb a finished, clean model’s inputs at inference time — the model itself is uncompromised. Defending against one does not defend against the other. Both matter, and they have separate defenses.

Frequently asked questions

What is the difference between data poisoning and a backdoor attack? Data poisoning is the broad category: any manipulation of training data to corrupt model behavior. A backdoor attack (also called a Trojan attack) is a specific, targeted subtype of poisoning where the attacker’s goal is to embed a hidden trigger — a specific token, pattern, or phrase — that causes a predictable malicious output while the model behaves normally on all other inputs. All backdoor attacks are poisoning attacks, but not all poisoning attacks install a backdoor. Some poisoning attacks aim for broad accuracy degradation or biased outputs without a specific trigger. The tutorials on backdoors and Trojaned AI models and adversarial examples cover these variants in detail.

Does poisoning only apply to models I train myself, or does it affect fine-tuning on a foundation model too? It applies to fine-tuning equally, and arguably more acutely. When you fine-tune a foundation model on a small custom dataset, each example in that dataset has high leverage over the model’s behavior on your target domain. A poisoning attack that would require thousands of samples in a pre-training corpus might require only dozens in a fine-tune file. If your fine-tune data comes from external sources — a vendor, a contractor, scraped content — apply all four defense gates to that data before you run the fine-tune job.

How is this addressed by the major AI security frameworks? Three frameworks converge on this risk. OWASP LLM04:2025 (Data and Model Poisoning) lists it as a top-ten risk with prevention strategies covering data lineage, anomaly detection, and adversarial testing. MITRE ATLAS v5.1.0 (November 2025) catalogs “Poison Training Data” (AML.T0020) and “Backdoor ML Model” (AML.T0018) as distinct techniques, with RAG Poisoning added in the Spring 2025 expansion. NIST AI 600-1 (the Generative AI Profile to the AI RMF) recognizes data poisoning as an Information Security risk and calls for provenance controls, supply chain integrity verification, and adversarial testing as explicit mitigations.

What is RAG poisoning, and is it data poisoning? Yes. RAG poisoning is data poisoning applied to the document store that a Retrieval-Augmented Generation pipeline indexes. Instead of poisoning a static training dataset, the attacker injects malicious documents into the live knowledge base. Those documents get retrieved in response to queries, and the LLM incorporates the poisoned content into its outputs — without any formal fine-tuning step. MITRE ATLAS added RAG Poisoning and False RAG Entry Injection as distinct sub-techniques in Spring 2025, reflecting how common this attack surface has become. The mechanism is the same; the intervention point shifts from the training pipeline to the document ingestion pipeline, and the same provenance and validation controls apply.

How do I know if a model I downloaded from a public registry is already poisoned? You largely cannot tell from behavioral testing alone, because sophisticated poisoning is designed to be dormant. This is why the AI supply chain problem has its own emerging practice: AI BOM (AI Bill of Materials) and model signing. The goal is to verify cryptographically that the model weights you downloaded match the weights the publisher signed — analogous to verifying a software package hash. Without that chain of custody, you are trusting the registry and the uploader. The tutorial on AIBOM and Model Signing covers the tooling in detail.

Where this fits in the series

Data poisoning sits at the very beginning of the ML lifecycle — stages S0 (Data) and S1 (Training) in the attack surface model introduced in The AI Attack Surface Explained. Before the model exists, before any inference happens, the attack has already been executed. That is what makes it categorically different from the inference-time attacks that dominate most AI security discussions.

The next logical step is understanding what happens when an attacker does not have access to your training data and must attack the finished model live: Adversarial Examples Explained covers how a single carefully perturbed input can flip a model’s label with high confidence. The targeted, trigger-based variant of data poisoning — where a specific token activates a hidden behavior — is covered in Backdoors and Trojaned AI Models. And if you want to see the full attack surface these topics live within, The ML Lifecycle Attack Surface maps all six handoff points where poisoning and other attacks can enter.

Browse all tutorials to follow the full Securing AI series.