Pretraining Data Can Be Poisoned through Computational Propaganda

Very interesting paper from UW and Allen Institute

Today’s paper: Pretraining Data Can Be Poisoned through Computational Propaganda. Graf et al. July 16, 2026. https://arxiv.org/pdf/2607.15267

One reason data poisoning has become a more serious AI security question is that modern language models are trained on corpora so large, heterogeneous, and opaque that nobody can really inspect them end to end. A frontier model may absorb text from Common Crawl, public code, forums, scraped articles, and countless other web sources, all filtered through automated pipelines that are designed for scale rather than close human review. That creates an obvious asymmetry: it is cheap to add text to the web, but very hard to know exactly what made it into training.

Unlike a lot of the literature out there, today’s paper does not argue that language models are vulnerable in some abstract theoretical sense. Instead, it claims that ordinary web infrastructure may already provide a realistic path for adversaries to smuggle malicious text into the vast corpora used for pre-training.

This core idea is captured in Figure 1 below. The authors trace an attack from an injected public comment on a webpage, through web crawling and text extraction, through deduplication and quality filtering, and finally into model behavior. The authors are shifting the discussion away from exotic insider attacks and toward something more mundane: if a site lets strangers post content, and if that content is folded into the page in a crawlable way, then the training pipeline may treat it as part of the web’s usable text.

What makes the paper especially useful is that it does not stop at a scary intuition. The authors introduce a measurement framework called HALFLIFE, which estimates the probability that an injected piece of content is not only posted, but actually survives the whole pipeline into a training set. They break the problem into stages: can the attacker place content on a relevant page, does the crawler capture it in plaintext, and does it make it through the filters that model builders already use? That is a much more serious way to study poisoning than simply proving that some malicious string can appear online.

The practical punchline shows up in Figure 2 below. Their estimated end-to-end inclusion rate for comment-based poisoning is only about 0.13%, which sounds tiny until you remember the scale of Common Crawl. At that rate, an attacker trying to land a few hundred poisoned documents in the final corpus might need to target on the order of one hundred thousand to one million upstream pages. That is a large operation, but it is not obviously out of reach for automated commenting systems, which is exactly why the paper lands as more than a curiosity.

The downstream results are also harder to dismiss than I expected. In Table 1, the authors show that even trace amounts of poisoned pretraining data can shift model preferences toward attacker-favored answers in controlled experiments. The base models show clear contamination across sizes, and supervised fine-tuning reduces but does not erase the effect. That is an important nuance. The paper is not claiming that every poisoned comment will create a dramatic sleeper-agent failure. It is showing that small upstream distortions can measurably bias later behavior, which is already enough to matter.

I also appreciate that the paper does not treat every conceivable injection vector as equally plausible. One of its better habits is to ask not just whether content can be placed somewhere on the web, but whether it survives the specific mechanics of crawling and extraction. That is where the work feels stronger than more speculative security arguments: it is attentive to the boring middle layers where many attacks either become real or quietly die.

That comes through most clearly in Figure 7, which appears in the appendix rather than the main body. The figure compares a page without JavaScript rendering to the same page with rendered ads, and it helps explain why programmatic advertising looks much less viable as a poisoning route. Even when the ads visually appear, they usually resolve inside cross-origin iframes that do not leak advertiser text into the DOM-based plaintext that crawlers extract. In other words, the paper is not just saying “the web is dangerous”; it is distinguishing between kinds of web content that actually enter training data and kinds that mostly do not.

In closing this musing, I think the paper makes a very interesting and empirically compelling point - there are clear results rather than just perspectives here. My main reservation is about generalization, not the basic finding. The authors use Common Crawl and a released filtering stack as a proxy for real production pipelines, and the exact numbers could change for labs with different crawl policies or proprietary curation systems. They also avoid live attacks for ethical reasons, which is the right choice, but it means the paper is strongest as a careful feasibility study rather than a census of how often this is already happening in deployed frontier models.

Even so, the central lesson seems hard to escape: once model builders rely on web-scale corpora they cannot fully inspect, content moderation and data governance become part of model security whether they like it or not. We live in interesting times.