Threat Modeling an LLM Application

A threat model is a short, written account of what a system holds, what it can do and who might push it somewhere it should not go. For an application built around a language model, the account is harder to keep accurate, because the model turns text into behavior in ways that a static design does not show.

Draw the data flow first

Start with a diagram of the parts and the lines between them. For a typical LLM application, the parts include the user interface, the application server that builds the prompt, the model call, any retrieval index, the tools the model can invoke, the backend systems those tools reach, and the logs. Draw the arrows and note what data moves along each one.

Then mark the trust boundaries. A boundary is a place where data passes from a zone you control to one you do not, or from one level of privilege to another. The usual boundaries are the user's input, content fetched from outside, the model's output when it drives an action, and the internal systems behind the tools. The model's output deserves its own boundary. If the application takes an action because the model asked for it, that output is effectively untrusted input to the tool.

The diagram does not need to be polished. It needs to be complete enough that a reviewer can point to any arrow and ask who controls the data on it.

List what would hurt

Write down the assets: the things that would cause harm if they were exposed, changed or made to behave wrongly. For an LLM application these often include customer records, internal documents, the credentials used by tools, the logic of the application's policies, and the ability to take actions such as refunds, bookings or messages. Reputation is an asset too. A system that gives confident wrong answers loses trust even when no data leaves it.

Be specific. "Customer data" is too vague to guide a control. "Order history for each customer, visible only to that customer and to support staff with an open case" gives a test you can run. The system prompt is also worth listing. Treat it as something a determined user may be able to coax out, and do not store secrets in it.

Name the actors and their goals

Next, list who could interact with the system and what each might want. An anonymous visitor might want data that is not theirs. A legitimate customer might want to act on another customer's account. An employee with access to the knowledge base might want to change what the assistant says. A third party whose document was indexed might want the assistant to behave differently when it reads that document. Someone holding a leaked key might want cheap access to the model.

Goals in these terms fall into a few groups: reading what they should not, taking actions they should not, changing what the system tells others, and consuming resources. The last group is easy to forget. Unbounded use of a metered model is a cost risk, and an assistant that can be steered into long loops can become a denial-of-service problem of its own.

Walk through a worked example

Consider a customer-support assistant. It reads a customer's order history, answers questions from help articles, and can issue refunds up to a limit set by policy. Its data flow runs from the chat interface to the application server, to the model, to the order service and the help index, and back.

Now list threats against that flow. If the help index is searched without filtering by customer, a question could return another customer's order details. If someone with editing access to the help articles writes text meant to change the assistant's behavior, the assistant could follow it when a customer asks an ordinary question. If the refund tool accepts any amount the model proposes, a single confused answer can exceed policy. If chat transcripts are stored in full, payment details may sit in a store with weaker controls than the payment system itself.

For each threat, write the control and say where it is enforced. The customer filter belongs in the retrieval query, not in the prompt. The refund cap belongs in the refund service, which rejects amounts over the limit whatever the model says. Refunds above a lower threshold should go to a person. Payment fields should be removed before a transcript is written.

Accept residual risk on purpose

Some risk remains after the controls. In the example, the model can still state a policy incorrectly, and no refund cap prevents a wrong explanation. Write that residual risk down, name its owner, and decide whether to accept it, reduce it further or add monitoring. A residual risk that nobody has named is a risk that nobody will watch.

Revisit the model whenever the design changes. A new tool, a new data source or a change in who can edit content each creates new paths, and the diagram is the place to see them. Update it with the change, not after the incident.

A short exercise

Pick an LLM feature you know, or a sketch of one. Draw its data flow in about ten minutes, marking the trust boundaries. List three assets and three threats. For each threat, write one control that is enforced outside the model, and name the component that enforces it. Then read the list and check whether any control would fail if the model did exactly what an attacker wanted. Those are the controls to strengthen first.