Skip to main content
Security guideReviewed

AI red teaming: what the term covers, and how to plan a test of your own AI application

AI red teaming means different things to standards bodies, security groups and model makers.

This guide sets their definitions side by side, separates testing a model from testing the chatbot or agent your business runs, and gives you a test-plan template, with a worked example, for the part you own.

AI application test plan template, with a worked tenant-assistant example

A plan to copy for testing a chatbot, assistant or agent your business deploys: threat hypotheses, permitted targets and exclusions, safety constraints, a register for each test case written before it runs, the evidence that sets severity, a person's review of every result and a retest status. The worked example is an invented rental agency, not a client system, and ends by setting an application test beside a network red-team exercise for the same business.

ai-red-teaming.txt

Introduction3 lines

AI APPLICATION TEST PLAN (VERSION 1) For a chatbot, assistant or agent that your business deploys on a model it does not train. Fill in Parts 1 to 5 before the first test message and have the owner of the application sign them off. Parts 6 to 8 are filled in as results come in. Write in the blanks; where a prompt sits in round brackets, replace it with your answer. Run the plan only against a deployment your business owns, or one whose owner has authorised the test in writing, and only within the usage terms of the model provider and any platform the deployment runs on.

What AI red teaming means, in the words of the bodies that use it

The bodies that publish on AI red teaming do not use the term in quite the same way, and the differences decide who should do the work.

Microsoft's guide to planning red teaming for LLMs explains that the term historically described systematic adversarial attacks for testing security vulnerabilities, and that with the rise of LLMs it has extended beyond traditional cybersecurity to describe many kinds of probing, testing and attacking of AI systems.

Anthropic, writing about the red teaming it uses on its own AI systems, points to a lack of standardised practices: developers might use different techniques to assess the same type of threat model.

Here is how five sources put it, each in its own terms.

NIST: a structured exercise to find flaws in an AI system

NIST's Generative AI Profile of its AI Risk Management Framework (NIST AI 600-1, July 2024) describes AI red-teaming as a structured testing exercise used to probe an AI system to find flaws and vulnerabilities such as inaccurate, harmful or discriminatory outputs, often in a controlled environment and in collaboration with system developers.

It calls the practice evolving, says it can take place before or after a model or system is made available to the public, and includes among its suggested actions AI red-teaming to assess resilience against attacks such as prompt injection.

It also says AI red-teaming results should be given additional analysis before they are incorporated into an organisation's governance, decision making and AI risk management.

OWASP: adversarial simulation against generative AI systems

OWASP's GenAI Red Teaming Guide (version 1.0, January 2025) describes GenAI red teaming as simulating adversarial behaviours against generative AI systems, such as large language models, to uncover vulnerabilities related to security, safety and trust. It covers both the AI models at the centre of an application and the systems used across the application's lifecycle, in four areas: model evaluation, implementation testing, infrastructure assessment and runtime behaviour analysis.

The guide notes that red teaming in cybersecurity has historically referred to adversarial simulations designed to test an organisation's defences across its IT assets, and says all stakeholders need to agree the specific scope and objectives of an AI red-teaming exercise to avoid miscommunication.

Microsoft: separate layers for the base model and your application

Microsoft's planning guide is written for responsible AI risks, such as harmful content in a model's output. It says an application built on a base model may need testing at several layers: the base model with its safety system in place, which is usually tested through an API endpoint; the application itself, best tested through its user interface; and both, before and after mitigations are in place. It adds that red teaming is not a replacement for systematic measurement.

Anthropic: a model maker testing its own models

Anthropic calls red teaming a critical tool for improving the safety and security of AI systems, and describes it as adversarially testing a technological system to identify potential vulnerabilities. The methods in its June 2024 post are ones it has used on its own AI systems, including frontier red teaming focused mainly on chemical, biological, radiological and nuclear risks, cybersecurity and autonomous AI risks.

It says the inconsistency between developers' practices makes it hard to compare the relative safety of different AI systems objectively.

NIST's glossary: the older meaning, a red team against an organisation

The traditional sense is still in use. NIST's glossary, quoting CNSSI 4009-2022, defines a red team as a group of people authorised and organised to emulate a potential adversary's attack or exploitation capabilities, with the objective of demonstrating the impacts of successful attacks and what works for the defenders, the blue team, in an operational environment. Nothing in that definition is specific to AI.

Three jobs that share the name, and who each one belongs to

Set side by side, those descriptions cover three different pieces of work.

They have different targets, different owners and different people who can fix what is found, so the first line of any plan should say which one it is.

Red teaming a foundation model

The target is the model itself: what it can be made to produce or do for anyone who uses it. Anthropic's frontier red teaming of its own models is this kind of work. So is the first of the four phases in OWASP's blueprint, which tests the model directly for toxicity, bias and alignment and for ways around any defences that may have been included in training, and reviews the security of the model's development life cycle.

The model maker controls the training, the weights and the built-in safety systems, so that is where these findings can be fixed.

A business calling a hosted model through an API chooses the model and its settings; it does not control how the provider trains it.

A red-team exercise against an organisation

The target is the organisation: its network, accounts, devices and, where the exercise's consent covers them, its people and premises. NIST's glossary definition describes this kind. OWASP's guide lists the traditional process as including threat modelling, reconnaissance, initial access, privilege escalation, lateral movement, persistence and exfiltration, and says traditional red teaming focuses on well-defined system compromises such as stealing domain administration credentials.

A chatbot may be one route such an exercise tries, but the exercise is judged on what it reaches across the organisation.

Adversarial testing of an AI application

The target is the deployment: the instructions a business wrote, the documents it connected, the tools it switched on, the systems those tools reach and what happens to each reply. OWASP's other three blueprint phases cover this ground. The implementation phase tests for ways around guardrails such as those in a system prompt and for poisoning of the data used to ground answers.

The system phase looks at how the model interacts with other components, including excess agency. The runtime phase targets business process failures and may assess the impact on downstream processes that consume generated content.

Microsoft's application layer is the same target, and it is the part a business that deploys a chatbot or agent can change.

LLM red teaming for a business that runs a chatbot or agent

For a customer chatbot, a staff assistant or an agent built on a hosted model, the useful version of LLM red teaming asks one question: what can this deployment be pushed into doing or revealing, and which layer stops it?

Every case in the plan should trace back to something the business controls.

  • The instructions the bot runs on, and what happens when a user argues with them.
  • Every channel that puts text in front of the model: the chat box, uploaded files, form fields, inbound emails, retrieved documents and tool results.
  • Each tool: whom it acts for, which arguments the model fills in, and what the server checks regardless of what the model asks.
  • The data the bot can read, including other customers' records and documents written for staff.
  • Where each reply goes next: a chat widget, a staff screen, an email or another system.
  • Usage limits on the model and on any tool that costs money per call.

What this kind of plan leaves out

The plan on this page contains no prompts aimed at getting around a model provider's safety systems or at producing harmful content. That work belongs with the model maker's own red teaming. A deployment's own rules, such as which records a customer may see or what the bot may promise, are tested with harmless markers instead: an invented reference code that should never appear in a reply, or a synthetic record that belongs to a different test account.

OWASP's prompt injection entry for 2025 says it is unclear whether fool-proof prevention methods exist, and its excessive agency entry recommends implementing authorisation in downstream systems rather than relying on the model to decide whether an action is allowed. That is why the plan grades results on what the systems behind the bot did.

The guides in this library on prompt injection, indirect prompt injection, RAG security, tool permissions, MCP servers and agentic workflows each go deeper on one channel; this page covers the plan that ties them into one bounded test.

AI red teaming methodology: what the plan settles before the first test message

The template on this page has eight parts, and each one answers a question that, left open, turns into an argument once the results are in.

OWASP's guide asks for the same discipline: each engagement needs clearly articulated objectives and explicit success criteria, and the scope should precisely define which models and systems will be tested, what types of tests will be conducted and what is explicitly excluded.

Threat hypotheses: start from what would hurt

Write each hypothesis as a sentence that begins "It would be bad if the bot..." and name the system that would have to act for it to happen. 1 min read

Permitted targets, named exactly

List each target as an address, an API endpoint or a named system, with its environment: a staging chat widget, the chat API behind it, each tool in its test mode, a test copy of the knowledge base and the screen where staff read transcripts. 1 min read

Safety constraints: what is never attempted

This part lists what the test will not do, whatever the results seem to invite: no real customer data in any input, no prompts aimed at harmful content or at a provider's safety systems, no messages to real people and no flooding. 1 min read

A register written before anything runs

Each test case goes into the register before its first run: the hypothesis it tests, the channel, the account, the exact input with its marker, how many runs it gets, the safe result expected and what counts as a failure. 1 min read

Evidence that sets severity

Rate each result on what the systems did, not on how alarming the reply reads. 1 min read

A person reviews every result

A named reviewer reads each result before it is recorded as a finding, an observation, no issue or inconclusive, and writes the reason beside it. 1 min read

Retest status in four outcomes

After fixes, the registered cases are run again on an identified build, and each finding is recorded as fixed, partially fixed, still reproducible or not retested, with the reason. 1 min read

LLM adversarial testing produces a rate, not a yes or no

The same input can produce a different reply on the next run, so each case is recorded as violations out of runs completed.

OWASP's guide calls this consistency testing: multiple attempts for each adversarial prompt, because a prompt that fails at first may succeed on a later attempt. For prompt injection it says a single successful adversarial response may indicate a vulnerability, with further testing to confirm whether it can be reproduced consistently.

That cuts both ways. One success in ten runs on a case that reaches another customer's records is a finding, not noise. Ten refusals in ten runs is a result for those runs on that build, not a promise about the next attacker or the next model version.

Cases that go straight to an API are different again: they test code that checks the session, so one run per variant shows whether the check holds, while a case that depends on what the model says needs several.

A run that errored, timed out or left no tool log is inconclusive. It is rerun once the logging works, and it does not count as a pass.

The worked example: a tenant assistant for an invented rental agency

The template's worked example follows one chatbot through all eight parts.

The agency, its tenants, addresses and systems are invented for this page as an illustration, and the results are made up to show the format. It is not a client system and not a record of a real test.

The agency manages rental homes. Its tenant assistant answers questions from the public about renting, drawing on a tenant handbook and the agency's website. A tenant who signs in can see their own rent ledger, lodge a maintenance request, check its status and ask for a call-back, which emails their property manager.

An outside developer built and hosts it, calling a model provider's API that bills by usage, and its tools reach the agency's property management software through that software's API.

Seven hypotheses come out of that description, from one tenant seeing another tenant's ledger to a reply in the staff transcript view that loads an outside image. Eight registered cases and a benign control test them on a staging copy connected to a sandbox portfolio, with two synthetic tenancies, a signed-out visitor and a staff reviewer.

The illustrative results show each kind of outcome: a fixed finding, a partially fixed one, one not yet retested and an inconclusive run.

What a network red-team exercise for the same agency would look at instead

The same agency could commission a network red-team exercise, and it would ask a different question. In the terms of NIST's glossary, it would emulate a potential adversary and show the impact of successful attacks and what works for the defenders. Its targets would be the office network, staff accounts and email, laptops and phones, and possibly staff themselves through phishing or phone calls under their own consent.

The chatbot might be one route it tries. The last block of the template sets the two exercises side by side, including where they meet.

Authorisation, provider terms and every system the bot can reach

A chatbot test reaches further than the chat box.

The tools call other systems, the knowledge base may sit with a search service, and every reply comes from a model provider. Testing needs the written authority of whoever owns the application, and of the owner of each system the tests touch.

NIST SP 800-115 makes the same point for networks: where part of an organisation's systems sits on a third party's network, the owner of that network usually must also consent in writing to the assessment plan.

The model provider's and any platform's own usage terms govern what may be sent to their services, during a test as at any other time. Read them before the plan is signed, and record in Part 1 of the template who read them and when. A test case that would not fit within them comes out of the plan. No plan can widen what those terms allow.

The people the bot talks to need the same care. Test accounts, synthetic records and test inboxes keep real tenants, customers and staff out of the results, and the stop condition on real personal information means any that does appear halts testing instead of ending up in the evidence.

This page's plan sits beside the general scoping documents in this library: the scope template, the rules of engagement and the authorisation letter.

Where Clever Ops fits, and who handles the rest

Clever Ops' AI chatbot and agent security test covers the application side of this page: a deployed bot or agent, from the chat box to the systems behind it.

That takes in the conversation, indirect channels, the retrieval pipeline, each tool and integration, MCP servers, the chat backend as an API, output handling and cost controls. A test copy with the same tools is preferred, with test identities and test records.

The report records the exact inputs used, the tool calls each attempt produced, how many attempts were made and how many succeeded, a CVSS v3.1 score with a separate business-risk rating, and a retest outcome for each finding.

  • Red teaming a foundation model, or testing the model provider's platform: Clever Ops does not do this work. How a base model behaves sits with the provider, so take it to the model provider or to a specialist in AI model evaluation.
  • Network red-team exercises, social engineering and physical testing: not part of any Clever Ops service. Refer them to a network or red team specialist.
  • Load, denial-of-service or volumetric testing: refer to a load-testing specialist, with the host's written approval.
  • A tender or contract that requires CREST: Clever Ops is not CREST accredited, so we refer that work to a provider that is.
  • A chatbot or agent Clever Ops built or hosts: the work is a security review rather than an independent test, and the report says who built it and who hosts it.

Bringing the plan to a scoping call

Bring Parts 1 to 3 of the template, filled in for your own bot, to the scoping call: what the bot is, who owns each system it reaches, and what would be bad if it went wrong.

Anything in your plan that falls outside the application side is listed in Part 3 with the kind of specialist it belongs to, so it is clear from the start what the test will not cover.

Frequently asked questions

Definitions differ by body. NIST's Generative AI Profile describes AI red-teaming as a structured testing exercise used to probe an AI system to find flaws and vulnerabilities such as inaccurate, harmful or discriminatory outputs. OWASP's GenAI Red Teaming Guide describes simulating adversarial behaviours against generative AI systems to uncover vulnerabilities related to security, safety and trust. In use, the term covers testing a foundation model, testing an application built on one, and sometimes a red-team exercise against a whole organisation, so a plan should say which it means.

They overlap. A test of an LLM application covers the web application and API around the bot as a penetration test would, and adds cases aimed at the bot's behaviour: instructions it can be argued out of, documents that steer it and tools it can be pushed into calling. OWASP's guide describes GenAI red teaming as combining traditional security testing methodologies with methods focused on the specific risks of generative AI. Because replies vary between runs, results are recorded as rates over repeated runs.

No. Clever Ops tests a business's own chatbot or agent deployment, not the AI model behind it, and does not test the model provider's platform. Red teaming a foundation model sits with the model provider or a specialist in AI model evaluation. Clever Ops also does not run network red-team exercises, social engineering or physical testing; that work goes to a network or red team specialist.

Yes, if it is your deployment and you work from a written plan: the owner's written authority, permission from the owner of each system the bot's tools reach, a check of the model provider's and platform's usage terms, a test copy where one exists, synthetic records with harmless markers, and spending limits in place. The template on this page is built for that. Do not point it at a chatbot your business does not own.

At minimum: what would be bad if the bot did it, exactly what may be tested, what must never be attempted, each test case written down before it runs, the evidence that sets severity, a person's review of every result, and a retest status for each finding. Those are Parts 2 to 8 of the template. OWASP's guide adds that rules of engagement should cover operational guidelines, safety controls and ethical boundaries.

Only if its rules say so. A network red-team exercise is judged on what an adversary could reach across the organisation, and a chatbot may be one route it tries. Testing what the chatbot itself can be pushed into, case by case with test accounts and tool logs, is a separate plan.

No. A rate of violations over registered runs describes those runs on that build. Microsoft's guide warns against reading specific red-teaming examples as a metric for how widespread a harm is, and says red teaming is not a replacement for systematic measurement. Use the results to decide what to fix and what to measure next.

Scope the right test for your application

Tell us what you have built, who uses it and when it needs to launch. You will get a plain-English recommendation: a website audit, a web application test or a referral to a specialist.