AI Security
AI red teaming: break your AI before attackers do
AI red teaming is testing an AI system the way an attacker would, to make it ignore its instructions, reveal what it should not or misuse its tools. It differs from a penetration test because the attack is written in plain language, and it ends with fixes you can retest.

Published 4 min read
AI red teaming is testing an AI system the way an attacker would: trying to make it ignore its instructions, reveal data it should not, or use its tools in ways nobody intended. It differs from a penetration test because the attack is written in plain language, so the weak point is often what the system is told and how it behaves, not only how it is built. A good test ends with each failure reproduced, ranked and fixed, then retested.
When AI systems do what they are told, by the wrong person
These are real cases, each from the company, tribunal or advisory that recorded it.
- A chatbot's promise became the company's. In February 2024, a Canadian tribunal held Air Canada responsible for what the chatbot on its website told a customer about bereavement fares, and ordered it to compensate the customer. What an AI says to a customer can bind the business.
- A colleague's message became an attack. In August 2024, Slack patched an issue in Slack AI where, in its words, a malicious actor with an existing account in the same workspace could phish users for certain data. Slack found no evidence of unauthorised access.
- An email nobody opened could leak data. In June 2025, Microsoft published CVE-2025-32711, an AI command injection in Microsoft 365 Copilot that let an unauthorised attacker disclose information, rated 9.3, critical. The researchers who found it, Aim Labs, named it EchoLeak and described it as zero-click. Microsoft fixed it on its side; users had nothing to do.
None of these needed a broken firewall. Each started with words the AI read and trusted. IBM's Cost of a Data Breach Report 2025 found that 13% of the 600 organisations studied reported a breach of an AI model or application, and that 97% of those had no AI access controls in place.
How it differs from a penetration test
A penetration test looks for flaws in how an application is built: its code, its configuration, its access control. Testing an AI system adds four differences.
- Language is the input. The attack is a sentence, a document or a web page, so it targets what the system is told and how it behaves.
- Results are not repeatable. The same prompt can work once in ten. Each attack is repeated, and a finding is a rate, not a yes or no.
- Harm is a finding, not only access. An AI that promises a refund it should not, or gives dangerous advice, has failed even if nothing was breached.
- The application still needs its own test. The login, the APIs and the access control around the model need an ordinary penetration test, which our red teaming and penetration testing service runs.
What we test first
The parts teams most often leave untested before release are the ones we start with. The OWASP Top 10 for LLM Applications, 2025 edition, names each of them.
- Indirect prompt injection (LLM01). Instructions hidden in content the AI reads, such as a document, an email, a web page or a support ticket, rather than typed by the user. EchoLeak worked this way, and our prompt injection examples show more.
- What the AI can do with its tools (LLM06, Excessive Agency). Sending email, changing records or calling APIs, often with more rights than the person using it.
- System prompt leakage (LLM07). Whether the instructions, keys or internal details written into the system prompt can be pulled out.
- Other people's data (LLM02, Sensitive Information Disclosure). Whether one user can get the AI to show another user's data, or internal documents it can search.
Each test is planned against known techniques: MITRE ATLAS, the public catalogue of attacks on AI systems, listed 129 techniques as of October 2026, and NIST's taxonomy of adversarial machine learning names the attacks and their mitigations. OWASP's GenAI Red Teaming Guide describes how such a test is run.
The mistakes we see most
Before a system is tested, the same four mistakes come up again and again:
- Secrets in the prompt. API keys, internal addresses or business rules written into the system prompt and treated as hidden. Anything in a prompt can be read back.
- Too much access. The AI connected to data and tools with broad rights to make it work, more than any user of it has.
- Trusting the output. The AI's answer used directly to act, to send, publish or change a record, with no check by code or a person.
- No logging. Prompts, tool calls and answers are not logged, so nobody can tell later whether the system was misused.
What you receive
At the end of our AI system security review, you receive:
- Every finding, reproducible. The exact prompts or content that triggered it, how often it worked, its severity and how to fix it.
- Fixes by layer. Tighter permissions on models and data, checks on what goes in and comes out, limits on what the AI may do on its own, and secrets moved out of prompts.
- A retest. Once your team has made the fixes, we test them again, so you know which issues are closed.
- What to watch in production. The events to log so an attack on the live system is noticed, the starting point for detection rules your security team runs, which our threat detection and SIEM service can build.
The review takes 2 to 6 weeks at a fixed price. Hardening, when the review calls for deeper changes, takes 2 to 4 months, at a fixed price or on time and materials.
Before you release
Ask four questions of any AI system before customers or staff use it. What can it read that a stranger could write? What can it do without a person approving? What would it reveal if asked cleverly? And would you know if someone tried? If any answer is "we are not sure", test it before an attacker does.
Frequently asked questions
Is AI red teaming the same as AI penetration testing?
The terms are used for the same work, testing an AI system the way an attacker would. Some tools sold as "AI pentesting" mean the opposite, using AI to test ordinary systems. This article, and our review, are about testing the AI system itself.
We use a vendor's model. Do we still need to test?
Yes. The vendor tests the model. You built the system around it, with your instructions, your data, the tools it can call and the people who can reach it. Most of the failures in this article sit in those choices, not in the model.
How long does an AI system security review take?
Our review of one AI system takes 2 to 6 weeks at a fixed price. If it calls for deeper changes, hardening the system with your engineers takes 2 to 4 months, at a fixed price or on time and materials.
Is AI red teaming required by law?
For most companies, not yet. The EU AI Act requires adversarial testing from the providers of general-purpose AI models with systemic risk, not from every company that deploys AI. Testing before release is still the only way to know how your own system behaves when someone tries to misuse it.
Sources
- Moffatt v. Air Canada, 2024 BCCRT 149, British Columbia Civil Resolution Tribunal, 2024
- Slack security update, August 21, 2024, Slack, 2024
- CVE-2025-32711, M365 Copilot Information Disclosure Vulnerability, Microsoft Security Response Center, 2025
- Cost of a Data Breach Report 2025, press release, IBM, 2025
- OWASP Top 10 for LLM Applications 2025, OWASP Foundation, 2025
- GenAI Red Teaming Guide, OWASP Foundation, 2025
- MITRE ATLAS data, version 2026.10, MITRE, 2026
- NIST AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST, 2025
AI governance consulting and AI security
We help you govern, secure and watch the AI systems your company builds or buys. That means a governance framework your leadership can stand behind, a security review that tests how each system can be misused, the hardening it needs, and detection your team can operate.
Explore AI SecurityRelated reading
AI Security
10 ways your AI can be turned against you: OWASP Top 10 for LLMs vs MITRE ATLAS
A chatbot that invents a refund policy, an assistant that leaks mail without a click, a coding tool shipped with planted code. The OWASP Top 10 for LLM Applications names these risks for builders; MITRE ATLAS records how attackers carry them out. Here is how the two fit together.
7 min read
AI Security
Agentic AI security: your newest employee has admin rights and no manager
AI agents do more than answer. They call tools, read data and act in other systems, often with a person's keys. Five public incidents from 2025 show what goes wrong, and each points to a control, from an identity of the agent's own to a named owner who can stop it.
7 min read
Tell us which AI systems you need to govern
List the tools and systems you use, built or bought. We reply within two business days.