Red Teaming an AI System: What to Test First

An AI feature that no one has tried to misuse has not been tested. It has only been observed on good days. Red teaming is the deliberate attempt to find the ways a system fails under pressure, and for AI features it starts with the system's reach rather than with clever wording.

Agree the scope and the rules first

Write the scope before any testing starts. Name the systems in scope, the accounts used for testing, the data that may be touched, and the time window. Record the stop conditions: the events that end a test at once, such as exposure of real personal data, an action taken against a production account, or an effect on a third-party system. Name who must be told when a stop condition is reached, and who receives the findings.

Get agreement from the owners of anything the feature connects to. If the assistant calls an internal API, that API's owner should know testing will happen, when it will happen, and how to tell a test call from an incident. Testing against real customer data is a different and far more serious activity. Avoid it unless it has been explicitly approved, and prefer synthetic records that look like the real ones.

A short written document is enough. Its purpose is to keep the test defensible if someone asks later what was done and why.

Start from what the feature can do

The most useful thing to know before testing is the feature's reach. List what it reads, what it writes and what it can call. A chatbot that answers from public documentation has a small reach. An agent with access to a mailbox, a ticketing system and a payments interface has a large one, and the same test plan will not serve both.

Order the tests by consequence. Reach into data the requester should not see matters most, then actions the requester should not be able to trigger, then the content the feature produces. A flaw in wording is real, but it is rarely as costly as a permission the feature did not need in the first place.

The four test categories

Organize the test cases into four categories, which together cover most of the ground.

Data exposure asks whether the feature returns information outside the requester's rights. That includes information belonging to other users, internal documents and the feature's own configuration. Test it with requests from accounts holding different rights, and check whether the answers respect those rights.

Unauthorized actions asks whether the feature can be led to use a tool in a way its policy forbids. Test each tool with arguments that policy should reject, and check that the rejection happens inside the tool, not only in the model's reply.

Unsafe content asks whether the feature produces output that is harmful, misleading or inappropriate for its audience. Most people think of this category first, and it matters, but for most business systems it carries less consequence than the first two.

Bypassed policies asks whether a rule still holds when it is enforced only in the prompt. A rule that exists only as text in the instructions can be overridden by other text. Test each such rule from several angles, including through documents and tool outputs, not only through the chat box.

For each category, vary the phrasing, the language, the format and the route in. A case that works in one wording and fails in another is still a finding.

Record findings so they can be fixed

A finding is useful only when someone can act on it. Record the category, the feature and configuration tested, the date, the model and prompt versions, the general nature of the approach, the observed effect and the consequence if the problem were exploited. Assign severity based on consequence and reach, not on how clever the approach was.

Keep detailed reproduction material in a restricted location, because a list of working approaches is itself sensitive. Give the developers enough to reproduce the problem and no more than they need. Report rates with their conditions. A single failure in a single run is a lead, while a set of runs under stated conditions is evidence.

Retest after every change

Fixes often move a problem rather than remove it. A rule added to the prompt may stop one phrasing and leave another working. A tool check may close one path while a document route stays open. Retest the same category with new variants after any change, and rerun the whole set after a model or prompt update.

Keep the test set versioned so that results from different times can be compared. Retire cases that no longer test anything, but keep a record of why they were retired. A set that only grows eventually becomes too slow to run, and a set that changes without notes cannot show progress.

Practice with a first test plan

Choose a feature in a sandbox you own, or a demonstration assistant built for the purpose. Write a one-page scope. Then list its reach and write ten test cases spread across the four categories. Describe what each case would check without writing any content that would work as an attack. For each case, state the result that counts as a pass and the result that would make you stop the test.

Give the plan to a colleague and ask them to run it without your notes. Where they cannot follow it, the plan is not yet precise enough to hand to a real team.