Prompt Injection: When Data Gives Orders

Suppose an assistant reads your inbox and drafts replies. One message in the pile is not from a person at all. It came from an automated form, and someone filled that form with text meant for the assistant to read, not for you.

Why instructions and data blur together

A language model receives its input as one stream of text. The instructions written by the developer, the message typed by the user, the documents retrieved to answer the question and the output of any tool all arrive as tokens in the same context. The model has been trained to follow instructions, which is much of its usefulness, so a passage inside a retrieved document that reads like an instruction has a real chance of being followed.

Compare this with a database. A parameterized query sends the code and the data separately, and the database never treats a value as a statement. Nothing equivalent exists for natural language. A developer can label content as data, and that label helps, but the model reads the label as more text. Treat any separator as a strong hint, not a guarantee.

This is why prompt injection is hard to fix at its root. The weakness comes from how these systems take input. Firmer wording in a system prompt reduces the problem without removing it.

Direct and indirect injection

Direct injection happens when the person typing to the assistant tries to override its rules. The attacker is the user, and the target is usually the application's own policy, such as a restriction on topics or a rule against revealing its configuration. The harm is often limited to what the assistant says, although it grows when the assistant has tools.

Indirect injection is more serious for systems that read external content. The text arrives through something the application fetched for a good reason: a web page, a shared document, an email, a support ticket, a file in a repository. The person who wrote that text may never talk to the assistant at all. They only need to get text in front of it, and the victim is the person who asked an ordinary question.

For agent-style systems, most of the concern sits in the indirect case. The attacker's reach is the set of places the application reads from, not the set of people who can log in.

What an injection can actually do

The consequence depends on what the assistant can do. An assistant that only produces text can mislead a reader, invent a source or reveal something placed in its context. That is a real problem, but its scope is the conversation.

An assistant that can send messages, change records, call an API or read private files can take actions on the strength of text it read. A second concern is output channels. Anything that can move data out of the system, including links or images that a client loads automatically, can carry information to a place the owner did not intend. Review each channel as a possible path for data, not only as a display feature.

The useful question for a review is therefore not whether the model can be tricked, since the answer is almost always yes. The useful question is what the worst reachable action would be once the model has been tricked. That answer sets the urgency.

Defenses that reduce the risk

No single control removes prompt injection, so the practical goal is to stack controls so that each limits the damage when another fails. The main layers are these:

• Give each tool the narrowest permission it needs, and keep read tools separate from write tools.
• Require confirmation for consequential actions, showing the exact action in plain terms before it runs.
• Validate tool arguments in ordinary code against policy, rather than trusting values the model produced.
• Mark retrieved content as untrusted in the prompt structure, knowing that this helps but does not guarantee the model will ignore instructions inside it.
• Restrict where output can go, for example by not auto-loading links to arbitrary domains.
• Log tool calls and watch for unusual sequences, so that a successful injection leaves evidence.

These controls work because they do not rely on the model's judgment. A model that resists a malicious document most of the time still fails on the occasions it does not, so enforcement has to sit outside it.

Testing for it

Test injection the way you test any control: with a defined set of cases, measured results and a record of what changed. Build a corpus of your own documents that contain instruction-like text asking for something harmless and easy to observe. Then check whether the assistant follows it, whether it reveals that it saw it, and whether any tool was called. Test indirect routes as well as direct ones, because indirect routes are the ones most often missed.

Report results with their conditions: which model configuration, which retrieval setup and which tools were enabled. A pass under one configuration says little about another. Keep the corpus under version control, so that a change to a prompt or tool can be checked against the same cases. Keep any test content that could work as a real attack in a restricted location, and do not treat a model refusing a test as proof that a tool cannot be misused.

Review exercise

Take a design for an assistant that reads external content, or a feature you already run. List every source of text the model reads, and mark which sources can be written by someone other than the user or the developer. For each marked source, trace which actions are reachable from the model's output, and note which of those would need a confirmation step.

Then pick the one path where a bad document could reach a consequential action with no human check. Write the control that would break that path. If the control depends on the model declining to act, rewrite it so that code outside the model enforces the limit.