The ML Lifecycle Attack Surface: 6 Handoffs, 6 Injection Points

June 25, 2026 · Securing AI: How AI Gets Attacked — and Defended (part 5)

▶ Watch on YouTube & subscribe to The Stack Underflow

Every team that ships an AI model draws the same diagram: raw data flows in on the left, a registered model comes out on the right, with a handful of boxes in between. It looks like a factory floor — clean, linear, controllable. But an attacker reads the same picture differently. They do not see the boxes; they see the arrows. Each arrow is a handoff between two systems, two teams, or two environments. Each handoff is a trust boundary. Each trust boundary is an injection point.

The ML build pipeline is not just how you construct a model — it is the attacker’s map of your entire pre-deployment surface. Understanding this is foundational: the later poisoning, backdoor, and supply-chain attacks in this series all live inside this one diagram, one handoff at a time.

The one-sentence version: Every step in the ML lifecycle hands something off to the next step, and every handoff is a place an attacker can inject — so the arrows between the boxes are the attack surface, not the boxes themselves.

The six-step build pipeline

The canonical build pipeline runs through six stages. The exact tooling varies by organization, but the logical sequence is consistent across virtually every ML project, from a small fine-tuned classifier to a trillion-parameter foundation model:

COLLECT --> LABEL --> TRAIN --> VALIDATE --> PACKAGE --> REGISTER
   (S0)      (S0)     (S1)       (S1)         (S2)        (S2)

• COLLECT — raw data is gathered from sources: web scrapes, licensed datasets, internal logs, public repositories.
• LABEL — data is annotated with ground-truth answers, either by human annotators, automated pipelines, or a mix.
• TRAIN — a model is fitted on the labeled data; weights are adjusted iteratively to minimize error.
• VALIDATE — the trained model is evaluated on held-out data; benchmarks, safety evaluations, and red-team tests run here.
• PACKAGE — the validated model is serialized into a distributable artifact (a checkpoint, a .safetensors file, a container image).
• REGISTER — the artifact is pushed to a model registry or artifact store where downstream systems can pull it for deployment.

The three stage labels (S0, S1, S2) correspond to the broader attack-surface ribbon introduced in The AI Attack Surface Explained. This episode covers the build half; the runtime half starts at S3.

The handoffs are the attack surface

Here is the key shift in perspective. Draw a trust boundary — a dashed perimeter — around the whole pipeline. Now notice where that boundary gets crossed: not inside the boxes, but on every arrow between them.

 OUTSIDE THE TRUST BOUNDARY
 ┌─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐

 |  [COLLECT]--handoff--[LABEL]--handoff--[TRAIN]--handoff--       |
 |  [VALIDATE]--handoff--[PACKAGE]--handoff--[REGISTER]            |

 └─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘
       ^inject        ^inject       ^inject               ^inject

Why does this matter? Because each handoff involves a moment when data or an artifact leaves the protection of one system and enters another. If that transition is not verified, it is a window: an attacker who can influence what is handed over controls what the next step receives, without ever touching the pipeline itself.

Provenance — the auditable record of where each artifact came from and what transformations it has undergone — is the property that closes this window. Without it, you cannot tell whether what you received is what was sent.

Attack by attack: what lands at each handoff

The table below maps each handoff to the primary attack, the framework identifier, and the mechanism. Each is covered in depth in later tutorials; the goal here is to name them so you can place them on the pipeline.

OWASP LLM04: Data and Model Poisoning (2025 edition) covers the first three rows — anywhere the data or weights themselves are contaminated. OWASP LLM03: Supply Chain (2025 edition) covers the last two rows — anywhere the artifact is tampered with or substituted in transit. MITRE ATLAS v5.1.0 (November 2025) maps specific techniques to each stage, including “Poison Training Data” and “Backdoor ML Model.”

The 2025 expansion of LLM04 from the narrower “Training Data Poisoning” to “Data and Model Poisoning” is significant: it acknowledges that the attack surface extends beyond the raw dataset all the way through fine-tuning, embedding generation, and weight modification at training time.

Three of the attacks in detail

Handoff 1 — Data poisoning at COLLECT

Data poisoning means introducing malicious or manipulated samples into the training dataset before the model ever sees it. Because ML models generalize from examples, a few carefully chosen bad examples can shift the learned decision boundary in ways that are hard to detect post-hoc.

The threat is concrete. In March 2024, JFrog’s security research team identified roughly 100 malicious models on Hugging Face with embedded code-execution payloads hidden in pickle serialization files (a well-known format for saving Python model weights). Python’s pickle format allows arbitrary code execution at load time — so loading a “model” can mean running attacker-controlled code. PickleScan, the primary open-source detection tool, was itself found to have three zero-day bypass vulnerabilities in December 2025 (CVE-2025-10155, CVE-2025-10156, CVE-2025-10157, each rated CVSS 9.3), meaning the detection layer was being neutralized.

Attacker-controlled source
        |
        v
[bad_sample_1, bad_sample_2, ...] injected into training corpus
        |
        v
   COLLECT --> LABEL --> TRAIN

The model trains on poisoned data, learns the wrong pattern, and passes validation if the evaluation set does not cover the poisoned region.

Handoff 3 — Backdoor injection at TRAIN

A backdoor (also called a Trojan trigger) is a hidden behavior baked into the model weights during training. The model behaves correctly on all normal inputs. Only when a specific trigger pattern appears — a particular word, pixel patch, or token sequence known only to the attacker — does the hidden behavior activate.

This is why evaluation evasion (handoff 4) pairs with backdoor injection: the backdoor is invisible to standard benchmarks if the benchmark never presents the trigger. The attack passes your test suite and ships.

Normal input  --> [Model] --> Correct output
Triggered input --> [Model] --> Attacker-controlled output

MITRE ATLAS v5.1.0 catalogues this as “Backdoor ML Model.” Defending it requires behavioral evaluation specifically designed to probe for trigger-response patterns — not just accuracy on a held-out set.

Handoffs 5 and 6 — Artifact tampering and supply chain substitution at PACKAGE and REGISTER

Even a perfectly clean model can be poisoned after training if the artifact is tampered with before deployment. Artifact tampering swaps or patches the serialized model file between VALIDATE and REGISTER. Supply chain substitution registers a malicious artifact under a legitimate name, so downstream systems pull a trojanized model believing it is the genuine one.

In February 2026, the Koi Security team discovered 341 malicious skills in the ClawHub agent skill registry — a supply chain compromise distributing Atomic Stealer malware. The pattern mirrors what has been documented on PyPI and npm for years, now applied to model and agent artifact repositories.

How to defend: gates on every handoff

The defensive model is straightforward to describe and requires discipline to execute: place a verification gate on every handoff arrow. The defense-in-depth principle applies — no single gate is sufficient; each one stops a different attack class.

 [COLLECT]
     |
     v gate: VET SOURCES (allowlist, content filters, dataset provenance records)
 [LABEL]
     |
     v gate: LABEL REVIEW (human spot-check, label-consistency audits, inter-annotator agreement)
 [TRAIN]
     |
     v gate: BEHAVIORAL EVAL + RED-TEAM (adversarial probing, trigger-search, not just accuracy)
 [VALIDATE]
     |
     v gate: SIGN ARTIFACT (cryptographic hash + signature before packaging)
 [PACKAGE]
     |
     v gate: VERIFY SIGNATURE AT REGISTRY (reject any artifact whose signature fails verification)
 [REGISTER]

The provenance thread is the through-line: a signed, auditable chain that proves what each artifact is and where it came from, from the first collected data record all the way to the deployed model. NIST AI RMF’s four functions — Govern, Map, Measure, Manage — provide the organizational structure for sustaining this chain: governance defines who can authorize a new training data source; mapping identifies where provenance breaks could occur; measuring tracks coverage of your verification gates; managing responds when a gate fires or an anomaly surfaces.

Use .safetensors format instead of pickle wherever possible: unlike pickle, safetensors does not execute arbitrary code at load time, which eliminates an entire class of artifact-tampering attacks at the package handoff.

Common misconceptions

• “The model file is just weights — it can’t be malicious.” Pickle-serialized model files execute arbitrary Python code at load time. Loading an untrusted checkpoint from a public repository without verification is equivalent to running an unsigned executable. The Hugging Face incidents of 2024–2025 are real-world proof. Always verify checksums, prefer safe formats, and scan with up-to-date tooling.

• “Validation accuracy is proof the model is clean.” A well-crafted backdoor is specifically designed to be invisible to standard evaluation. If your evaluation set does not include the attacker’s trigger pattern, a backdoored model passes 100% of your tests. Behavioral security evaluation — adversarial probing, red-teaming — is a separate discipline from accuracy measurement and is not optional.

• “Only external attackers poison training data.” Insider threat is a well-documented vector in ML pipelines. A data engineer with write access to the training store, a labeler with a commercial incentive to flip annotations, or a poorly scoped CI/CD service account can all act as the injection point. Provenance controls need to cover internal actors, not just external ones.

• “Signing the final model artifact is enough.” Signing protects the PACKAGE-to-REGISTER handoff, but a backdoor baked in at TRAIN will survive signing and verification with its signature intact. Defense must be layered across all six handoffs, not applied at the end only.

Frequently asked questions

What is the difference between data poisoning and a backdoor attack? Data poisoning is the broader category: it means manipulating the data used at any stage of the ML lifecycle to influence the model’s behavior. A backdoor is a specific type of poisoning outcome: the model learns a hidden trigger-response association during training, so it behaves normally on all standard inputs but misbehaves only when the trigger is present. You can poison data without planting a backdoor (for instance, to degrade accuracy or introduce bias), but all backdoor attacks require some form of poisoning.

Why is the PACKAGE-to-REGISTER handoff considered a supply chain risk? Because at that point the model artifact becomes a dependency — something downstream systems pull and trust by name. This is structurally identical to a compromised npm package or a poisoned PyPI distribution. OWASP LLM03 (2025) classifies it here because once an artifact is in a registry under a legitimate identifier, any consumer of that registry is at risk. The attack does not require access to your training pipeline at all; it only requires write access to the registry or the ability to intercept the upload.

Can I defend the pipeline without slowing down the development cycle significantly? Most of the gates described here can be automated into CI/CD. Source allowlisting and provenance recording happen at data ingestion. Artifact signing is a one-command operation in tools like Sigstore or in-toto. Signature verification at registry pull can be enforced by policy without human intervention. The operationally expensive gate is behavioral red-teaming, which does require investment — but it only needs to run at training time, not on every inference.

Does this apply to fine-tuning an existing model, not just training from scratch? Yes, and in some ways the risk is higher for fine-tuning. Fine-tuning typically uses a smaller dataset, which means each training example has a larger influence on the final weights — making label-flipping and backdoor injection easier to execute with less data. The same six handoffs apply: the “COLLECT” stage is where the fine-tuning dataset is assembled, and all downstream handoffs carry the same risks. OWASP LLM04 (2025) explicitly covers fine-tuning and embedding data, not only pre-training.

How does MITRE ATLAS relate to OWASP LLM Top 10 for this topic? OWASP LLM Top 10 describes risk categories from the perspective of what can go wrong and what to prioritize. MITRE ATLAS (v5.1.0, November 2025) is a structured adversary knowledge base: it maps specific attacker tactics and techniques, including “Poison Training Data” and “Backdoor ML Model,” to the kill-chain stage where they apply. The two frameworks are complementary — OWASP tells you what to fix, ATLAS tells you how an adversary executes the attack. Using both together gives you a more complete threat model.

Where this fits in the series

This tutorial is the pipeline-level map. Every attack tutorial in the series points back to one handoff on this diagram. The next logical step is understanding what the attacker actually does to the data: Data Poisoning Explained goes deep on the mechanics of corrupting a training set, and Backdoors and Trojaned AI Models covers trigger-based attacks in detail. For the supply chain end of the pipeline, The AI Supply Chain and Securing MCP extends the PACKAGE and REGISTER handoffs into deployment.

If you want to understand where this pipeline sits inside the full six-stage attack surface, start with The AI Attack Surface Explained. For a structured approach to threat-modelling across the whole pipeline using ATLAS, OWASP, and NIST AI RMF together, see How to Threat Model an AI System.

Browse all tutorials to follow the full Securing AI series from substrate to governance.