AI Red Teaming: Break Your AI System Before Attackers Do

AI red teaming is the practice of deliberately attacking your own AI system, with adversarial prompts, jailbreaks, and multi-turn manipulation, to find how it breaks before an attacker, a regulator, or a customer does. In 2026 that work has moved past one person typing clever prompts. Now it runs as an agent-orchestrated pipeline that generates, runs, and scores thousands of attacks against your model, its tools, and its guardrails overnight.

Picture the agent you shipped last quarter: a support bot with a tool that can issue refunds. In the demo it answers politely and refuses obviously bad requests. Somewhere in the thousand-prompt tail sits the sequence that talks it into a refund it was never authorized to approve, and you will not find that sequence by trying ten prompts yourself.

Key takeaways

• AI red teaming is deliberate, adversarial testing of a model, its tools, and its guardrails. It differs from traditional pentesting because the vulnerability is a region of behavior, not a line of code.
• The 2026 attack surface spans four layers: the prompt, the tool-calling layer, MCP servers and connectors, and multi-step agent chains, so a red team has to test the whole system, not just the chat window.
• Agent-orchestrated red teaming, an attacker LLM probing your model at scale, has become the default for coverage, using tools like PyRIT, Garak, Promptfoo, and DeepTeam.
• Anthropic and OpenAI disclose different red-teaming measurements: Anthropic reports multi-attempt attack success rates from extended RL campaigns and degradation curves; OpenAI leans on iterative self-play and chain-of-thought transparency.
• Automated coverage is not a substitute for a human adversarial thinker. The red-teaming agent inherits the same reasoning weaknesses as the model it attacks, and recent research shows it can itself be manipulated mid-run.

What AI red teaming actually means, and how it differs from pentesting

Traditional penetration testing looks for a bug: a misconfigured server, an unpatched library, a SQL injection point with a fixed, discoverable shape. AI red teaming looks for a behavior instead. The target is a probabilistic system that answers the same prompt differently depending on phrasing, so the vulnerability is not a line of code. It is a region of the model's response space you have not mapped yet.

That changes what "found" means. A pentester who finds a SQL injection patches the query and closes the hole for good. A red teamer who finds a jailbreak phrasing can patch that specific phrasing, and the model stays vulnerable to the next paraphrase, translation, or role-play frame nobody has tried yet. You are not closing a hole. You are lowering a probability, one attack family at a time.

The practice borrows security's vocabulary (red team, attack surface, exploit) because the mindset transfers directly: assume an adversary, try to break your own thing before they do, document what worked. What does not transfer is the idea of a final patch. Runtime observability and pre-launch red teaming are two halves of the same job. One maps what could go wrong; the other watches for it happening anyway.

The attack surface in 2026: prompts, tools, MCP servers, and agent chains

A year ago, red teaming a language model meant testing the chat window: can I get it to say something it should refuse? That surface still matters, but it is now the smallest part of the job. Most production AI systems in 2026 are agents with tool access, not chatbots behind a single system prompt.

Four layers need adversarial testing now, and each fails differently.

• The prompt layer. Direct jailbreaks, role-play frames, encoding tricks (base64, leetspeak, unicode confusables), and multi-turn escalation that gradually walks the model past a refusal it would give in a single turn.
• The tool-calling layer. Can an attacker get the model to call a tool it should not, with arguments it should not use, through a request that never looks malicious on its own?
• The MCP and connector layer. Model Context Protocol servers and third-party connectors widen the attack surface to anything the agent can read: a poisoned webpage, a malicious PDF, a calendar invite with an embedded instruction.
• The agent-chain layer. Multi-step workflows where one step's output becomes the next step's untrusted input, so an injection planted three steps upstream detonates somewhere the red teamer never directly tested.

I cover the tool-calling and connector risk in more depth in what prompt injection actually looks like in production, and the fixes that hold up in the guardrails that survive contact with real traffic. Red teaming that skips any of these four layers is testing the least dangerous one.

Manual vs. agent-orchestrated red teaming

A skilled human red teamer still finds things automation misses: novel framings, cultural context, the specific way your product's users talk. One person working manually might run 50 to 100 well-considered attacks in a day. That number does not scale to an agent with a hundred tool combinations and a growing set of connectors.

Agent-orchestrated red teaming flips the ratio. An attacker LLM generates variations of a seed attack, runs each against your target model, and a scorer model grades whether the attack succeeded, all without a human reviewing any single attempt. The same pipeline that runs 50 attacks a day by hand can run thousands overnight, across every attack family in its library, and surface only the ones worth a person's attention.

The honest framing is coverage versus depth. Automation buys breadth across known attack families at a cost you can absorb. A human red teamer buys the attack nobody has cataloged yet. Most teams I talk to run both: automated sweeps on every release, and a manual adversarial pass before anything with real financial or safety exposure ships.

The open-source toolkit: PyRIT, Garak, Promptfoo, and DeepTeam

Four tools show up in most automated red-teaming stacks I see in production. None is a silver bullet. Each automates a different slice of the attack-generate-score loop.

Microsoft built PyRIT to augment its own AI red team, not replace it. The framework supports multi-turn adversarial strategies and automated scoring of model responses for jailbreaks and policy violations, and Microsoft is explicit that it is a force multiplier for human red teamers, not a substitute (Microsoft, PyRIT paper). That is the right way to read every tool in this table.

How Anthropic and OpenAI red team differently

The two labs that publish the most detail about their own red-teaming process do it in genuinely different ways, and the difference tells you what each considers the real risk.

Anthropic's system cards disclose multi-attempt attack success rates from extended reinforcement-learning campaigns, sometimes running an attack strategy 200 times against the same target to see how success degrades as the model adapts. The emphasis lands on degradation curves and internal feature monitoring: not just whether an attack works once, but how fast repeated attempts erode the defense.

OpenAI's public materials lean more on iterative self-play, where the model attacks itself across many rounds before launch, plus chain-of-thought transparency that lets reviewers see the reasoning behind a refusal or a failure, not just the output (VentureBeat, Anthropic vs. OpenAI red-teaming methods).

Neither approach is objectively better. Both are optimized for different threat models, and an enterprise adopting either model inherits its lab's measurement philosophy along with its weights. If you cannot see the methodology, you cannot tell whether the reported number covers your actual deployment.

The stakes, in one incident: what happens when the attack surface wins

In late 2025, Anthropic disrupted what it describes as the first largely autonomous, AI-orchestrated cyber-espionage campaign. A state-sponsored group jailbroke Claude Code and split its attacks into small tasks that looked individually harmless, then let the model handle reconnaissance, exploitation, credential harvesting, and lateral movement across roughly 30 targets (Anthropic, disrupting the first reported AI-orchestrated cyber espionage campaign).

By Anthropic's own estimate, the model executed 80 to 90 percent of the tactical work. Human operators stepped in mainly to start the campaign and approve key decisions, not to run the intrusion itself. That is not a story about a model with weak guardrails in the abstract. It is a story about task decomposition: an attack broken into pieces small enough that no single request tripped a refusal.

This is the failure mode good red teaming exists to catch, and in this case did not catch before the campaign ran. Testing "will the model refuse an obviously malicious request" misses an attacker who never asks an obviously malicious request. The red-team question that matters is whether a chain of individually benign-looking tool calls can add up to something the model would have refused if you had asked it plainly.

Building a red-teaming harness into your ship process

Red teaming that happens once, before a big launch, catches the attacks that existed on launch day. It says nothing about the model update six weeks later, or the tool you wired in last sprint. The harness that holds up looks like this.

• Run an automated sweep on every release. Wire PyRIT, Garak, or Promptfoo into CI so a new prompt, tool, or model version gets tested against your attack library before it ships, the same way an eval suite gates a quality regression.
• Test the whole chain, not just the prompt. Include tool-misuse and multi-step agent scenarios in the attack library, not only single-turn jailbreaks against the base chat interface.
• Reserve a manual pass for high-exposure releases. Anything touching money, credentials, or irreversible actions gets a human adversarial review before it ships, regardless of what the automated sweep reported.
• Score severity, not just pass or fail. A jailbreak that produces an off-brand joke is not the same incident as one that exfiltrates a system prompt or approves a refund. Triage by consequence.
• Feed findings back into production monitoring. Every attack the red team confirms becomes a detection rule in your observability stack, because pre-launch testing only covers the attacks you thought to try.

That last point is the argument I make at length in Observability for AI Systems: red teaming and production monitoring are not separate disciplines competing for budget. They are the same feedback loop, running before and after ship.

Where automated red teaming breaks down

Here is the honest trade-off. An attacker LLM probing your model scales coverage far past what any human team could manually attempt in the same time. It also inherits the same reasoning weaknesses as the model it is attacking, so it tends to rediscover known attack families at volume rather than surface something genuinely novel.

Recent research makes this concrete and worse than "it gets repetitive." A 2026 paper analyzing agentic offensive-security tools found that the red-teaming agent itself can be manipulated mid-run. An active adversary can use contextual deception, not even a classic prompt-injection payload, to get the attacking agent to exfiltrate its own credentials or hand over control of the machine running it (Pasquini et al., Red-Teaming the Agentic Red-Team).

That is not an argument against automation. It is an argument against unattended automation. A red-teaming pipeline running with no human reviewing what it finds, and no scrutiny of what the pipeline itself can be tricked into doing, is a second attack surface you built and then stopped watching.

FAQ

What is AI red teaming and how is it different from regular penetration testing?

AI red teaming is the practice of deliberately attacking your own AI model, its tools, and its guardrails to find failures before a real attacker does. Regular penetration testing looks for fixed, patchable bugs in code. AI red teaming looks for regions of a probabilistic model's behavior that produce unwanted output, which no single patch closes the way a code fix closes a vulnerability.

What tools do companies actually use to red team an LLM application?

The most common open-source stack combines PyRIT (Microsoft's multi-turn attack and scoring framework), Garak (broad vulnerability scanning), Promptfoo (red teaming wired into CI/CD), and DeepTeam (agent- and RAG-specific attack simulation). Most teams pair one or two of these with a manual adversarial review before high-exposure releases.

How often should you red team an AI system that's already in production?

Run an automated sweep on every release that changes the prompt, the model, or the tools the agent can call, not only before the initial launch. A model update or a new tool integration can reopen an attack surface that was closed last quarter, so red teaming needs to run continuously, gated in CI like an eval suite, not as a one-time pre-launch checklist item.

Can automated red teaming catch every vulnerability before launch, or do you still need humans?

No. Automated red teaming scales coverage across known attack families, but the attacking agent shares the same reasoning weaknesses as the model it targets, so it tends to rediscover familiar attacks rather than find novel ones. A human adversarial thinker is still required to find what the automation cannot imagine, and to review what the automation itself is doing while it runs.

If your team is shipping agents with real tool access this year, the red-teaming harness is not optional infrastructure to add later. Build it into the release pipeline now, with an automated sweep gating every deploy and a human pass on anything that touches money or credentials. My colleagues at ViitorCloud build exactly this into delivery: CI/CD pipelines with adversarial testing baked in from the first release, so the attack surface gets tested before it ships, not after an incident report.