Data Poisoning in Plain Language

Consider an internal assistant that answers staff questions from a shared drive. Its answers are only as trustworthy as the files it is allowed to read, and the people who can edit those files are a much larger group than the people who build the assistant.

Two places data gets in

Data can shape a model's behavior at two points. The first is training, including fine-tuning, where examples change the model's weights. The second is retrieval, where documents are indexed and then placed into the model's context when a question arrives. Retrieval is usually easier to change and quicker to affect, because indexes are rebuilt often and the content they contain is edited by many people. Training influence is harder to exploit at scale and harder to trace afterward, since its effects are spread across the weights rather than sitting in a document.

The distinction matters for controls. A retrieval source can be inspected, versioned and reviewed document by document. A training effect often cannot be inspected in the same way, so the defense has to start earlier, at the data that goes in.

A quieter route is feedback. When generated answers are saved back into a wiki or a ticket system and then indexed, the system's own output becomes one of its sources. Errors can then repeat and appear to be independently confirmed, because they now sit in several documents. Mark machine-written content in the source, and keep it out of any collection that is used as evidence for the system's own answers.

What poisoning means in general terms

Poisoning means that someone with influence over a data source deliberately shapes what a system learns or retrieves, so that its answers change in ways the owners did not intend. The change can be broad or narrow. A broad change degrades quality across many topics and tends to be noticed. A narrow change applies only to a particular topic or phrasing, which makes it harder to find and easier to hide among normal updates.

Content that looks authoritative is a common vehicle, because systems tend to trust well-formed, official-looking text. Labels can also be shifted, so that a classifier learns a wrong association. The common thread is that the influence arrives through a route the owners consider routine, such as an edit to a shared page or a batch of labeled examples from an outside contributor.

Provenance as the first control

Start by recording where each document or example came from, who can write to its source, when it entered the system and through which process. A source with broad write access and no review is the largest risk in the system, whatever its content looks like today.

Provenance records are only useful when they stay attached to the data. A hash kept in a spreadsheet that is not linked to the index it describes records nothing in particular. Restrict write access to sources that feed indexes or training sets, and require review for changes to the ones that matter most. Keep trusted and untrusted sources in separate collections, and mark the source of each retrieved passage so that the person reading an answer can judge it. Showing the source also acts as a check on the system. If an answer cites a page nobody expected, the mismatch is visible.

Review and monitoring

Review changes to high-impact sources the way you review code. A diff of what changed, who changed it and why is more useful than a full re-read after the fact. After each index rebuild, run a fixed set of questions with known answers and compare the results. These canary questions are cheap, and a change in their answers is an early signal.

Monitor retrieval itself. A document that is retrieved far more often than its neighbors, or that appears for questions it has no relation to, deserves a look. Track answer drift on the fixed test set over time. None of these signals proves poisoning, but each one points to where a person should read.

The limits of detection

Detection will not catch everything. A poisoned document can be written to look ordinary, and statistical checks mainly catch clumsy changes. Weight-level effects from training are hard to inspect, and checking a large model for a narrow change is an active research area rather than a solved problem. Treat detection tools as partial aids, and be skeptical of any claim that a system is clean because a scan passed.

The strongest remedies are structural. Keep versioned copies of datasets with their hashes, so you can roll back to a known state. For training-time problems, retraining from a clean dataset may be the only complete fix, and it is costly, which makes provenance controls worth more than cleanup. Discuss these trade-offs with the owners of the system before a problem forces the choice.

A provenance audit exercise

List every data source your assistant reads, both for retrieval and for any training or tuning. Next to each, write who can edit it, how changes are reviewed and how often it is rebuilt. Then identify the source with the widest write access and the weakest review. That is usually where the first control should go.

Finally, write twenty canary questions whose answers you can verify, and decide how often they will run and who reviews the results. A canary set that nobody reads is decoration, so give the review an owner and a schedule.