Red Teaming an AI System: Finding the Holes Responsibly

June 25, 2026 · Securing AI: How AI Gets Attacked — and Defended (part 32)

▶ Watch on YouTube & subscribe to The Stack Underflow

You ship an AI-powered feature. You run unit tests, you review the code, maybe you even run a static analysis scan. And then you hand it to users and quietly hope nobody discovers that the model leaks data under adversarial pressure, or that a crafted multi-turn conversation bypasses every guardrail you built. That hope is not a security posture.

AI red teaming — the practice of attacking your own system, on purpose, with written authorization, before a real adversary does — is how you replace hope with evidence. It is not a single tool or a one-liner. It is a structured process: plan, execute, and report responsibly. The stakes are real: research published in 2025 attributed roughly 35% of confirmed AI security incidents to prompt-based attacks, and estimated global losses from AI prompt-injection alone reached approximately 2.3 billion dollars, a 340% year-over-year increase. The gap between systems that were tested and systems that were not tested is where that number lives.

The one-sentence version: Red teaming is authorized, structured, adversarial testing of an AI system — the discipline of finding the holes on the board so they get patched before someone finds them in production.

Rule zero: authorization is not optional

Before a single test case runs, the exercise must be authorized. This sounds obvious, but it is the most commonly skipped step in informal AI testing, and skipping it turns a security engineer into a liability.

Authorization means a documented scope agreement — a written statement of what systems are in bounds, what access the red team has, what techniques are permitted, and what the rules of engagement are. Without it, you are not red teaming; you are attacking.

┌─────────────────────────────────────────────────────────┐
│  AUTHORIZED · SCOPE + RULES OF ENGAGEMENT               │
│  ✓ Systems in scope (named, versioned)                   │
│  ✓ Access level (black-box / gray-box / white-box)       │
│  ✓ Permitted techniques (listed by category)             │
│  ✓ Duration and reporting obligations                    │
│  ✓ Signed by system owner + security lead                │
└─────────────────────────────────────────────────────────┘
         ↑
    Every exercise passes through this gate first.
    No scope document = no test.

The three access levels matter because they change what you can find:

NIST AI 600-1 (the Generative AI Profile companion to AI RMF 1.0, published July 2024) defines AI red teaming as “a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with developers of AI.” That collaboration framing is intentional: the best red teams work with, not against, the development team. The goal is finding and fixing, not winning.

The six-step exercise structure

A red-team exercise is not a freestyle attack session. It follows a repeatable process that produces findings you can act on.

PLAN                             EXECUTE            REPORT
  │                                 │                  │
  ▼                                 ▼                  ▼
1. Set objectives                5. Run test cases   6. Disclose
2. Define threat model              (injection,         responsibly
3. Select techniques (ATLAS)        fuzzing,
4. Choose tools                     multi-turn)

Step 1 — Set objectives

The objective answers two questions: what are you trying to break, and what does a successful find look like? Vague objectives produce vague results. Good objectives are specific:

• “Can an unauthenticated user extract the system prompt via multi-turn conversation?”
• “Does the RAG pipeline retrieve and surface documents the requesting user should not see?”
• “Can a crafted user message cause the agent to call a tool with attacker-controlled parameters?”

Each objective maps directly to a risk in the OWASP LLM Top 10 2025 or a case study in MITRE ATLAS. Starting there means you are testing for known, documented failure modes rather than improvising.

Step 2 — Define the threat model

Pull up the attack-surface map — the S0-through-S5 ML lifecycle stages covered in The AI Attack Surface Explained. Decide which stages you are simulating, and which adversary persona you are embodying:

This framing matters because it stops the test from being a random walk. A red team simulating an external API attacker should not spend three days auditing training data pipelines — that is a different adversary with different access.

Step 3 — Select techniques mapped to MITRE ATLAS

MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) is the definitive framework for AI-specific attack techniques. Think of it as MITRE ATT&CK, but for machine learning systems. Version 5.1.0 (November 2025) contains 16 tactics, 84 techniques, 56 sub-techniques, and 42 documented case studies. Between October 2025 and February 2026, the framework expanded substantially to cover agentic AI patterns, with more than 60 techniques either net-new or rewritten for agent contexts.

Techniques follow the format AML.T#### (techniques), AML.TA#### (tactics), AML.M#### (mitigations). Selecting techniques from ATLAS rather than improvising has a concrete benefit: your findings report can map each finding to a framework ID, making it immediately actionable for both the fix team and any compliance audit.

MITRE ATLAS v5.1.0 — example technique selection

Tactic: ML Attack Staging (AML.TA0001)
  Technique: LLM Prompt Injection (AML.T0051)
    Sub-technique: Indirect Prompt Injection (AML.T0051.000)

Tactic: Exfiltration (AML.TA0010)
  Technique: LLM Data Extraction (AML.T0057)

Tactic: Evasion (AML.TA0004)
  Technique: Prompt Obfuscation (AML.T0054)

Each selected = one test category in the exercise plan.

Step 4 — Choose tools

Tools fall into three categories. The episode script is explicit: list them as categories, not as weapons. The point of this step is to select the scale of the exercise — hand-typed tests find some things; automated harnesses find many more.

Microsoft PyRIT, released as open source and battle-tested across 100+ internal operations, is the most widely referenced open framework as of 2025. It adapts its prompts based on model responses — effectively a feedback-driven fuzzer for generative AI. It was integrated into Azure AI Foundry in May 2025 as the AI Red Teaming Agent preview.

Step 5 — Execute the test cases

Three test types cover the bulk of what matters for an inference-layer exercise. They are processes, not payloads:

TEST TYPE 1: Prompt Injection Tests
  Input:  crafted user message designed to override instructions
  System: the AI application under test
  Check:  does the model follow the injected instruction instead of its system prompt?

TEST TYPE 2: Fuzzing
  Input:  high volume of mutated, boundary, and encoded variants
  System: same application
  Check:  does any input produce a policy violation, data leak, or unexpected tool call?

TEST TYPE 3: Multi-Turn Probes
  Input:  sequence of benign-looking messages that accumulate toward a policy boundary
  System: same application + any session memory
  Check:  does the model drift from its constraints across a long conversation?

The discipline here: document every test case that passes (does not find anything) as much as every one that fails. A clean run on 2,000 fuzzing cases is evidence. Undocumented testing is not.

What to do when you find a hole

A finding is not a moment of triumph — it is the start of a documentation task. Record it defensively:

• What: the precise behavior observed
• Where: the component, endpoint, or stage where it manifests
• Impact: what an attacker could achieve (data exfiltration, prompt override, tool misuse)
• Reproduction: the minimum sequence of steps that reliably triggers it — kept private

Do not share a working reproduce anywhere public, in a Slack channel, or in a GitHub issue. The reproduction evidence exists for the fix team, not as a demonstration.

How to disclose responsibly

Responsible disclosure is the protocol that separates a security researcher from an attacker. The flow is linear and time-bound:

FIND      →  DOCUMENT  →  REPORT      →  FIX        →  VERIFY   →  CLOSE
(private)     (private)    PRIVATELY       WINDOW        THE FIX     (public
                           to owner        (agree on              timeline
                                          deadline)              optional)

Key principles:

• Report to the system owner first, not to the press, Twitter, or a public disclosure database.
• Agree on a fix window before the engagement starts. Ninety days is a common standard for non-critical findings; critical findings with active exploitation risk often use a shorter window with coordinated public disclosure.
• Verify the fix before closing. A patched finding that re-appears in the next release is a reporting failure, not a development failure.
• If the owner fails to respond within the agreed window, escalate through a coordinating body (CERT/CC, for example) rather than going directly public.

The EU AI Act, which reached full compliance requirements in August 2026, explicitly mandates that high-risk AI systems have documented testing and incident-reporting obligations. Responsible disclosure practices are no longer just good ethics — for systems in scope of the Act, they are a compliance requirement.

How to defend using what red teaming finds

Red teaming’s output is a prioritized backlog of confirmed weaknesses. Translating that into defense is a four-step loop:

1. Triage by ATLAS technique ID. Each finding maps to an ATLAS mitigation (AML.M####). The mitigation entry tells you what control class addresses the technique — guardrails, input validation, output filtering, access control.
2. Patch the highest-impact findings first. Impact is not just severity of the finding; it is the product of severity and likelihood given your deployment context.
3. Add a regression test. The test case that found the hole becomes a permanent entry in your automated test suite. Every deployment runs it.
4. Re-test after patching. A finding is not closed until the exact reproduction steps no longer succeed against the patched system.

This loop — red team finds, defender patches, re-test confirms — is the feedback mechanism that keeps a threat model honest over time. The threat model you built in How to Threat Model an AI System is only accurate until the system changes. Red teaming is how you keep it current.

Common misconceptions

• “Red teaming is what you do once before launch.” Continuous red teaming matters more than pre-launch exercises. The system changes: new model versions, new tools, new context. A finding that did not exist last quarter may exist today. Treat red teaming as a recurring practice, not a milestone.
• “Automated tools cover everything; manual testing is optional.” Automated fuzzers find high-volume, pattern-based issues. Multi-step reasoning exploits, social-engineering-style prompt chains, and context-window manipulation often require a human tester who understands the application’s business logic. Both are necessary.
• “If I red team my system and find nothing, it is secure.” A clean result means the red team did not find anything with the techniques they used and the access they had. It does not mean no vulnerabilities exist. Scope limitations, time constraints, and technique selection all bound what a red team can discover. Clean results increase confidence; they do not guarantee safety.
• “Responsible disclosure only applies to external researchers, not internal teams.” Internal red teams still produce findings that could be damaging if mishandled — a leaked reproduction case can be as dangerous as a public advisory. Apply the same documentation and access-control discipline internally as you would expect from an external researcher.

Frequently asked questions

What is the difference between AI red teaming and traditional penetration testing? Traditional penetration testing focuses on network, application, and infrastructure vulnerabilities — think SQL injection, authentication bypass, and exposed services. AI red teaming targets the model’s behavior: does it follow instructions it should not, leak data it was trained on, or allow its tools to be hijacked? The techniques differ (prompt engineering versus network enumeration), the frameworks differ (MITRE ATLAS versus MITRE ATT&CK), and the findings differ (behavioral policy violations versus CVEs). Many engagements need both, run as separate workstreams with coordinated reporting.

Do I need a dedicated AI red team, or can my existing security team do this? Existing security teams can and should run basic AI red-team exercises, especially with tools like PyRIT that lower the technical barrier. However, deep behavioral testing — multi-turn probes, RAG poisoning validation, agent tool-misuse chains — benefits from someone who understands both adversarial ML and the application’s business logic. Many organizations start with their existing security team running ATLAS-mapped test suites and bring in specialists for white-box or advanced exercises.

How do I scope an AI red-team exercise without it expanding indefinitely? Scope creep is the most common failure mode for first exercises. Fix it at Step 1 (set objectives) and Step 2 (define threat model) by listing the specific OWASP LLM Top 10 2025 categories you are testing and the specific ATLAS tactics you are covering. Time-box the execution phase to a defined sprint. Everything outside the named scope is a finding for the next exercise, not this one.

Can red teaming get me into legal trouble if I find something serious? Without written authorization, yes — testing a system you do not own without permission can violate the Computer Fraud and Abuse Act (in the US) or equivalent statutes elsewhere. With written authorization scoped to systems you are authorized to test, no. This is why the scope document from Rule Zero is load-bearing: it is your legal protection as much as your operational boundary.

What frameworks should a report cite? A well-formed AI red-team report cites the MITRE ATLAS technique IDs for each finding, references the corresponding OWASP LLM Top 10 2025 category where applicable, and maps mitigations to ATLAS AML.M#### entries. For systems that fall under NIST AI RMF, map findings to the Measure function subcategories in AI RMF 1.0 and to the 12 risk categories in NIST AI 600-1. This traceability makes the report immediately useful to compliance teams, not just security teams.

Where this fits in the series

This tutorial is the first episode in the “defending at scale” arc. The attack surface you mapped in The AI Attack Surface Explained is the terrain the red team walks. The threat modeling discipline in How to Threat Model an AI System produces the threat model that scopes the exercise at Step 2. The specific attack techniques covered throughout the series — prompt injection in Prompt Injection: The Attack Flow Every AI Developer Must Know, RAG attacks in RAG Security: Why Your RAG Pipeline Retrieves a Backdoor, and agent failure modes in Multi-Agent Failure Modes — are the techniques you select at Step 3. Red teaming is not a separate discipline; it is the discipline of applying everything in this series to your own system, with authorization, on purpose. Browse all tutorials to follow the full series.