Why AI Is Uniquely Attackable: The Four Properties That Break Security

June 25, 2026 · Securing AI: How AI Gets Attacked — and Defended (part 4)

▶ Watch on YouTube & subscribe to The Stack Underflow

You already know how to secure software. Read the code, find the bug, ship a patch, write a signature. That playbook has held up for decades. Now drop a machine learning model into your architecture and try to apply it. You cannot read why the model decided what it did. You cannot patch a learned behaviour — you have to retrain the whole thing. The same prompt can produce different outputs on back-to-back calls, so your hash-based detection never fires. And the training dataset is so large that a poison needle is invisible in the haystack.

This is not software with new bugs. AI has four structural properties that defeat the security playbook itself — not just individual controls, but the underlying assumptions those controls depend on. If you can name these four properties and explain the attack class each one enables, every later attack in this series stops being magic and starts being predictable.

The one-sentence version: AI models are learned surfaces, not written rules — and because you can’t read, patch, fingerprint, or fully inspect a learned surface, every standard security tool finds itself swinging at air.

The difference that starts everything: code vs. a learned surface

Before the four properties, one conceptual distinction unlocks all of them.

Traditional software is a rulebook you wrote. Every decision traces back to an explicit if statement, a function call, a branch you can read in a diff. Security tools that work on software — code review, static analysis, patch management, signature detection — all depend on that readability.

A trained machine learning model is something else. Nobody wrote its rules. It learned a decision surface from data: a high-dimensional boundary that separates “spam” from “not spam,” “malware traffic” from “normal traffic,” “unsafe output” from “safe output.” A decision is not “which if statement fired?” It is “which side of the boundary does this input land on?”

  TRADITIONAL SOFTWARE              TRAINED MODEL
  ──────────────────────            ─────────────────────────
  if risk_score > 0.8:              [ billions of weights ]
      block_request()                    │
  else:                               decision surface
      allow_request()                    │
                                    "which side?"

  You wrote this. You can read it.  Nobody wrote this. You can't.

That one shift — from explicit rules to a learned surface — is the root cause of all four properties below.

Property 1: Uninterpretability — you can’t audit it like code

Uninterpretability means that even after a model is trained and deployed, you cannot fully explain why it produced a specific output for a specific input.

When you review code for a security issue, you read it. You follow the data flow, trace the call stack, look for the vulnerable branch. That technique assumes there are lines to read. A neural network with millions of parameters distributes knowledge across all of them simultaneously. There is no single node that “decides” — the decision emerges from the aggregate of millions of weighted calculations. Visualising those weights (as in tools like Netron) shows you a wall of floating-point numbers, not auditable logic.

The security consequence is direct: a backdoor can survive a review that finds nothing wrong, because there is nothing readable to find. MITRE ATLAS v5.1.0 (November 2025), which catalogs adversarial tactics against AI systems, includes model backdooring as a concrete technique precisely because the opacity of weights makes tampering invisible to standard inspection.

CVE-2024-3568, disclosed in April 2024, is a real example of how this property amplifies risk. The Hugging Face Transformers library loaded model checkpoints using Python’s pickle module without sanitizing the source. An attacker could embed arbitrary executable code inside a serialized model file that looked — to every scanner — like a normal checkpoint. The file passes inspection, loads silently, and executes. The root cause is not just the deserialization bug; it is the fact that “what this model does” is opaque, so nobody thought to verify the weights themselves (NVD, CVE-2024-3568, 2024).

What the attacker gains: the ability to hide malicious behaviour inside a model that passes every code-level review.

Defender’s response: because you cannot read the model, you must evaluate its behaviour. Behavioral testing — running the model against a curated set of trigger inputs and checking whether outputs shift — is the analogue of code review for learned systems. More on this in the defense section below.

Property 2: Unpatchability — there is no hot-fix

In traditional software, you find a vulnerable function, write a two-line patch, push the update, and the fix is live in minutes. The unpatchability of AI models means this flow does not exist.

A learned behaviour is not stored in a line of code. It is distributed across every weight in the network, shaped by every example the model ever trained on. To change the behaviour, you have to change the training — which means collecting or curating new data, retraining, re-evaluating, re-testing, and redeploying. That pipeline takes days to weeks, even for a well-resourced team. For most production systems, the realistic timeline for a full retrain-and-redeploy cycle after a vulnerability is discovered is measured in weeks, not hours.

  TRADITIONAL VULN RESPONSE         AI VULN RESPONSE
  ──────────────────────────        ──────────────────────
  identify bug                      identify problematic behaviour
       │                                 │
  write patch (hours)               curate/clean training data (days)
       │                                 │
  test patch (hours)                retrain model (days–weeks)
       │                                 │
  deploy (minutes)                  re-evaluate on test suite (days)
       │                                 │
  done                              regression test (days)
                                         │
                                    redeploy (hours)
                                         │
                                    done (weeks later)

The NIST AI Risk Management Framework 1.0 — with its four functions of Govern, Map, Measure, and Manage — exists partly to address this reality. Because you cannot hot-fix a model, governance has to be baked in before the model ships, not bolted on after an incident. NIST AI 600-1 (the Generative AI Profile, published July 2024) extends this to large language models, mapping pre-deployment testing and incident disclosure directly to the framework’s functions (NIST AI 600-1, 2024).

What the attacker gains: a structural window of exposure that is far longer than in traditional software. An adversary who discovers a model vulnerability has days to weeks of undisturbed access before a fix can reach production.

Defender’s response: because you cannot patch quickly, you defend in depth around the model — input validation, output filtering, monitoring — so that a compromised model cannot do unconstrained damage while the retrain cycle runs.

Property 3: Nondeterminism — signatures and hashes never match

Traditional security tooling — antivirus signatures, hash-based blocklists, intrusion detection rules — all share one assumption: the same bad thing produces the same fingerprint. A piece of malware hashes to a known value. A malicious payload matches a known pattern.

Nondeterminism breaks this assumption. Machine learning models, and large language models in particular, are stochastic by design. Inference involves sampling from probability distributions; temperature settings, nucleus sampling, and other decoding choices mean that the same input can legitimately produce different outputs on consecutive calls. The model does not have a fixed fingerprint for “this input.”

  input: "summarise this document"
       │
       ├──► call 1: "The document covers three main topics..."
       ├──► call 2: "Three key themes emerge from the text..."
       └──► call 3: "This text explores the relationship between..."

  sha256(call 1) != sha256(call 2) != sha256(call 3)

  [SIGNATURE FILTER: no match for any known-bad hash]
  [RESULT: attack traffic passes undetected]

This is not a hypothetical problem. Researchers at Palo Alto Networks demonstrated this in a published study: by manipulating HTTP header fields in traffic that a deep-learning-based malware detector was trained to classify, they caused the model to misclassify malicious command-and-control traffic as benign — on every variation they tested — because the nondeterministic nature of the model’s learned boundary meant no two outputs matched the same rule (MITRE ATLAS case study, Palo Alto Networks C2 evasion).

OWASP LLM Top 10 — 2025 edition lists nondeterminism as a compounding factor in multiple risk categories, including LLM01:2025 (Prompt Injection) and LLM10:2025 (Unbounded Consumption), because the variable nature of outputs makes both attack detection and abuse-rate limiting structurally harder.

What the attacker gains: evasion of any defense that relies on matching a fixed pattern. Every variant of a malicious input is a new fingerprint that the blocklist has never seen.

Defender’s response: because you cannot sign or hash outputs, you must evaluate behaviour statistically. Anomaly detection on output distributions, rate limiting on token consumption, and semantic similarity checks (not string matching) are the right tools here.

Property 4: Scale — poison hides in the haystack

The final property is the one most likely to be underestimated. Scale means two things in AI security:

1. Data scale: modern models train on billions of examples. You cannot inspect them by hand, and even automated scanners cover a tiny fraction of the surface.
2. Parameter scale: the largest language models have hundreds of billions of weights. The attack surface is not a function or a file — it is a high-dimensional space that no human or tool can exhaustively explore.

  TRAINING DATASET (billions of rows)
  ┌─────────────────────────────────────────────────────┐
  │  row 1    row 2    row 3    ...    row 1,000,000,000 │
  │                                                     │
  │                         [POISONED ROW hidden here]  │
  │                                                     │
  └─────────────────────────────────────────────────────┘
        ↑
  Manual inspection covers ~0.000001% of this.
  Automated scan misses semantically plausible poison.

OWASP LLM Top 10 — 2025 edition tags this directly as LLM04:2025 (Data and Model Poisoning). MITRE ATLAS v5.1.0 maps it to the “AI Attack Staging” and “Execution” tactic phases — an attacker who can influence even a small fraction of training data can implant behaviour that generalises across the entire deployed model.

A real incident that illustrates scale as a vector: the alleged theft of training data and model techniques associated with DeepSeek R2 (reported 2025). The claim — still unverified — was that proprietary training data and fine-tuning techniques were exfiltrated from OpenAI. If accurate, the attack surface was not a single API endpoint; it was the entire data and model artifact pipeline, spanning billions of training examples and model checkpoints (Practical AI Security, Farlow, 2025; multiple press reports). The economic consequence was immediate: US AI company stock prices dropped sharply on news of a rival model trained at a fraction of the legitimate cost.

What the attacker gains: the ability to hide malicious behaviour in a place no defender can exhaustively check, and to do so before the model ever ships.

Defender’s response: because you cannot inspect everything, you track provenance — where did this data come from, who handled this model artifact, what was the chain of custody? Data signing, model signing, and AI Bill of Materials (AIBOM) documentation are the practical tools.

The four properties mapped to the attack surface

These four properties do not exist in isolation. They map directly onto the build pipeline that produces an AI system:

None of these properties is a bug someone introduced carelessly. They are structural features of how machine learning works. That is what makes them so important to understand: you cannot fix them by writing better code. You contain them by building the right defenses around them.

How to defend against each property

The key framing is contain, don’t cure. You are not going to make a deep neural network interpretable or patchable by rewriting it. You build a defensive posture that accounts for those constraints.

Against uninterpretability — evaluate behaviour, not code:

• Build a behavioral test suite: a curated set of inputs (including adversarial probes) that you run against every model version before and after any change.
• Use red-teaming to find behavioural failures that code review cannot find. MITRE ATLAS v5.1.0 includes red-team case studies from Microsoft’s AI Red Team as direct examples.
• Treat model artifacts (weight files, checkpoints) as untrusted binaries — scan them, sign them, and verify signatures before loading. CVE-2024-3568 is the cautionary tale.

Against unpatchability — govern before you ship:

• Follow the NIST AI RMF 1.0 Govern and Measure functions: define acceptable behaviour before deployment, not after an incident.
• Build monitoring that can detect behavioural drift in production — because when a vulnerability is discovered, you need to contain it at the application layer while the retrain cycle runs.
• Maintain a staged rollback: keep the previous model version deployable so you can revert while a fix is being prepared.

Against nondeterminism — use statistical and semantic detection:

• Replace string matching with anomaly detection on output distributions. Track token counts, topic drift, refusal rates, and sentiment shifts over time.
• Apply rate limiting at the semantic level (e.g., detecting that many queries are semantically similar to known jailbreaks) rather than exact-match blocklists.
• Log all inputs and outputs to a tamper-evident store so that unusual patterns are discoverable in retrospect even if missed in real time.

Against scale — track provenance end to end:

• Maintain an AIBOM (AI Bill of Materials): a structured inventory of every dataset, pretrained checkpoint, and library version used in the model’s training pipeline. OWASP LLM Top 10 — 2025 edition lists supply chain risk (LLM03:2025) as a top-tier concern.
• Cryptographically sign training datasets and model artifacts at each handoff. An unsigned artifact at deployment is a gap in the chain of custody.
• Apply statistical data validation before any external data enters the training pipeline: check for distribution shift, label anomalies, and unusual token patterns.

  DEFENSIVE POSTURE ACROSS THE BUILD PIPELINE
  ─────────────────────────────────────────────────────────
  S0 DATA            S1 TRAIN           S2 MODEL
  ───────────        ────────           ────────
  [provenance]  -->  [evaluation]  -->  [behavioral test]
  [signing]          [red-team]         [monitoring]
  [validation]       [governance]       [output filtering]

  Assume inputs are hostile.
  Track everything you cannot inspect.
  Evaluate behaviour you cannot read.
  Monitor in production what you cannot hot-fix.

Common misconceptions

• “AI security is just AppSec with a new target.” AppSec tools assume readable, patchable, deterministic code. All four properties above violate those assumptions. You need AppSec plus AI-specific practices — behavioral testing, provenance tracking, statistical monitoring — not AppSec instead of them.

• “If the model passes its test suite, it is secure.” A test suite is a finite sample. Because of scale and uninterpretability, a model can behave correctly on every test case while containing a backdoor that only activates on a specific trigger input not present in your suite. Test suites are necessary but not sufficient; red-teaming and adversarial probing are required to find what you did not know to look for.

• “Nondeterminism means attacks are unreliable, so attackers will prefer traditional targets.” Nondeterminism makes defenses harder, not attacks. An attacker who wants to evade a signature-based detector benefits from nondeterminism — every variant of a malicious prompt is a new fingerprint the blocklist has not seen. The attacker does not need the attack to work 100% of the time; they need the defense to miss it.

• “The model vendor is responsible for these properties.” The model vendor can improve robustness, interpretability, and documentation. But the gating, provenance tracking, behavioral monitoring, and architectural isolation are the responsibility of the team deploying the model. A perfectly trained model deployed without output filtering, input validation, or monitoring is still a security liability.

Frequently asked questions

Why can’t we just make models interpretable and solve the problem at the root? Interpretability research is advancing — mechanistic interpretability, SHAP values, attention visualization — but current techniques provide partial and approximate insight into large models, not the line-by-line auditability of source code. Even interpretable-by-design models (decision trees, linear classifiers) trade off accuracy for transparency in ways that make them impractical for many production tasks. The realistic posture today is to use interpretability tools as one layer of a defense-in-depth strategy, not as a replacement for behavioral testing and monitoring.

What does MITRE ATLAS cover that ATT&CK does not? MITRE ATT&CK catalogs adversary tactics and techniques against traditional software and network infrastructure. MITRE ATLAS v5.1.0 (November 2025) extends this specifically to AI and ML systems, adding tactics with no ATT&CK equivalent — including AI model access, AI attack staging, and techniques like data poisoning, model evasion, and model extraction. ATLAS currently documents 16 tactics, 84 techniques, and 42 real-world case studies, and the November 2025 release added 14 new agent-focused techniques in collaboration with Zenity Labs.

How does unpatchability affect incident response timelines? In traditional software incident response, a critical vulnerability might be patched and deployed within hours. For an AI model, the equivalent timeline — data remediation, full retrain, evaluation, regression testing, staged deployment — is typically measured in days to weeks. This means that when an AI vulnerability is disclosed, the window during which attackers can exploit it is structurally longer. The practical implication: your incident response plan for an AI system must include application-layer containment measures (e.g., disabling the affected feature, adding output filters) that can be deployed in hours while the longer retrain cycle runs.

Is nondeterminism a property of all AI models or just large language models? All stochastic machine learning models exhibit nondeterminism to some degree — it appears in training (random initialisation, dropout, stochastic gradient descent) and, for generative models, in inference (sampling from output distributions). Classical ML models like random forests are also nondeterministic across training runs. For LLMs, nondeterminism at inference time is particularly pronounced because it is a feature, not a bug: temperature and sampling parameters are tunable dials that trade output diversity against determinism. Setting temperature to zero makes LLM outputs more reproducible but does not eliminate all sources of variation, and it changes the model’s usefulness for many tasks.

Does the EU AI Act change any of these technical realities? The EU AI Act (enforcement beginning 2025–2026, depending on risk category) does not change the underlying properties of machine learning — a model is still uninterpretable, unpatchable, nondeterministic, and large regardless of regulation. What the Act does is impose governance obligations that force organisations to document, test, and monitor AI systems in ways that partially address these properties. High-risk AI systems must maintain technical documentation, undergo conformity assessment, and implement risk management processes that map closely to the NIST AI RMF functions. The Act creates legal accountability for the consequences of these properties, which is a meaningful external driver for doing the security work described above.

Where this fits in the series

This tutorial sits at the foundation of the series. The four properties — uninterpretability, unpatchability, nondeterminism, and scale — are the reason every specific attack covered later works the way it does. Data poisoning works because of scale and uninterpretability. Adversarial examples work because of nondeterminism and the nature of the learned decision surface. Backdoors survive because of uninterpretability. Slow response to model-level vulnerabilities is a consequence of unpatchability.

Start with The AI Attack Surface Explained for the full six-stage pipeline map that this tutorial’s properties attach to. Then see Why Signature-Based Defenses Fail on AI for a deeper treatment of nondeterminism’s effect on detection tooling, and Data Poisoning Explained to see the scale property turned into a concrete attack. If you are building threat models, How to Threat Model an AI System shows how ATLAS, OWASP, and MAESTRO map onto these properties in practice.

Browse all tutorials to follow the full series.