Pretraining Data Can Be Poisoned through Computational Propaganda

Image generated by AI

TLDR

• What: Attackers can poison web-scale LLM pretraining data by injecting adversarial content into public discussion/comment interfaces (e.g., WordPress), bypassing modern HTML extraction and curation filters.
• Who's at risk: Foundational language models (such as OLMo, DCLM, and FineWeb) that train on uncurated web crawls without strict user-generated content (UGC) filtering.
• Key number: Public comment injections have an estimated end-to-end inclusion probability of 0.13%, meaning an attacker only needs to automate injections across 100k to 1M web pages to successfully implant backdoors or bias LLM pretraining.

As foundational language models scale past trillions of tokens, the security boundary of pretraining shifts from highly curated datasets to the messy, unvetted expanse of the open web. Modern LLMs like DCLM and OLMo rely heavily on automated web scrapes (such as Common Crawl) which are highly vulnerable to adversarial manipulation. This post details how malicious actors can exploit public discussion interfaces (like WordPress comments) to inject computational propaganda directly into pretraining datasets—bypassing standard deduplication, language identification, and quality filters.

Background / Problem Setup

Previous work on pretraining data poisoning has operated under narrow threat models. For instance, Carlini et al. (2024) targeted established data sources like Wikipedia or purchased expired domains. However, Wikipedia represents only a tiny fraction (0.067%) of a modern pretraining corpus, making it unrepresentative of overall pretraining data distribution. Furthermore, prior attacks have largely ignored the crucial interaction between poisoned data and raw web extraction/curation pipelines.

The paper introduces HALFLIFE, a novel analytical framework that measures whether injected content actually survives web crawling, text extraction, and heuristic filtering to reach the final training corpus.

Methodology

The HALFLIFE framework decomposes the end-to-end probability of poison inclusion, $P(\text{include} \mid v, \mathcal{S})$, into three sequential stages for a given attack vector $v$ and data pipeline $\mathcal{S}$:

1. Webpage Injectability ($S1$)

To estimate how much of the web is injectable, the authors scanned 181,857 web pages sampled from Common Crawl shard CC-MAIN-2025-51 (Section 4.1). They detected public comment platforms on 3.4% of pages:

• $P(\text{injectable}) = 0.034$
• WordPress dominated the landscape, accounting for 85.2% of comment-bearing pages (Table 3).

2. Scraped and Extracted Text ($S2$)

Even if a comment is on a page, it must survive HTML extraction. LMs do not train on raw HTML; they use text extraction tools like Resiliparse (used in Dolma 3 and DCLM) to discard boilerplate, navigation, and ads. The authors found that user-submitted comments generally appear in static HTML plain text:

• $P(\text{captured} \mid \text{injectable}) = 0.719$ (71.9% of injected comments survive extraction).

3. Survival Through Data Curation ($S3$)

Extracted text must pass through three curation stages (Section 4.3):

1. Heuristic filtering (length, word length, repetition, and symbol-to-word ratios).
2. Language filtering (FastText language identification retaining English probability > 0.65).
3. Quality filtering (FastText quality classifier scoring page text).

Surprisingly, 5.5% of captured, injectable adversarial comments survived all filtering stages ($P(\text{not filtered} \mid \text{captured}, \text{injectable}) = 0.055$). For comparison, natural comments survive at 7.2%, meaning the adversarial text only slightly reduces survival and does not drastically disrupt document-level quality scores.

Combining these stages yields an end-to-end inclusion probability of 0.13% for public comment injections.

Why Programmatic Ads Fail as a Poisoning Vector

The authors used HALFLIFE to analyze programmatic ads as an alternative vector (Section 4.6). They found ads to be completely unviable:

• Static scrapers (like Common Crawl) capture empty HTML placeholders (92.9% of units) because they do not execute client-side JavaScript.
• Rendered scrapers (using headless browsers like Playwright) load ads, but 75.9% resolve to cross-origin iframes shielded by the browser's same-origin policy, rendering them invisible to DOM-based text extraction.

Key Results

To evaluate the downstream impact of comment-based pretraining poisoning, the authors trained a model ladder (65M, 150M, 260M, 709M, and 1.3B parameters) on a subset of the Dolma 3 dataset, injecting varying trace amounts of poisoned tokens (0.1%, 0.01%, 0.001%).

They tested three injection formats:

1. USER/ASSISTANT (explicit chat transcript style)
2. Q/A (generic question-answer markers)
3. NO-LABEL (naturalistic prose with no markers)

Table 2 shows the change in model preference ($\Delta$) for the attacker-preferred entity (e.g., favoring Boeing over Airbus) compared to a clean baseline:

A Skeptical Look at the Data

• Base Model Vulnerability: Base models are highly vulnerable across all parameters, showing persistent preference shifts of ~17% to 20% at a 0.1% token poison rate.
• SFT Cleansing Effect: Supervised Fine-Tuning (SFT) aggressively washes out the poisoning effect on larger models. For the 1.3B model, SFT reduced the USER/ASSISTANT shift from +19.0% to a marginal +2.6%, and completely nullified the NO-LABEL format (-0.3%).
• The Persistence of Q/A: While SFT diminishes the attack, the generic Q/A format remains competitive at 1.3B, retaining a +2.2% preference shift. Stealthier, more naturalistic formats are harder for standard instruction-tuning to clean out.

Limitations & Open Questions

While the paper presents a highly rigorous framework, several limitations must be considered:

1. In-the-Wild Anti-Bot Measures: The authors used sandboxed WordPress environments to validate automated posting. In production environments, attackers would face server-side defenses (such as rate limiting or platform-level moderation tools) that would heavily depress $P(\text{injectable})$.
2. SFT Decay on Larger Models: The sharp drop-off in poison retention after SFT as models scale (from +9.0% at 65M down to -0.3% at 1.3B for NO-LABEL) suggests that for much larger models, the retention rate of low-volume comment injections might fall significantly.
3. Common Crawl as a Proxy: Production pipelines at major AI labs may differ in scope, frequency, and filtering compared to raw Common Crawl, and they likely deploy proprietary text extraction and filtering methods.

What Practitioners Should Do

If you are curation-engineering or developing pretraining data pipelines, implement the following mitigations:

1. Comment-Aware Text Extraction

Standard text extraction tools (like Resiliparse or Trafilatura) do not distinguish between primary page content and user comments. Configure your extraction pipelines to strip elements matching known comment class signatures before parsing.

# Conceptual example using BeautifulSoup to strip comment blocks
from bs4 import BeautifulSoup

def strip_comments(html_content):
    soup = BeautifulSoup(html_content, 'html.parser')
    # Target common WordPress, Blogger, and generic comment classes/IDs
    comment_identifiers = [
        "comment-content", "wp-block-comment", "comment-list", 
        "comments-area", "respond", "facebook-comments"
    ]
    for identifier in comment_identifiers:
        for element in soup.find_all(class_=lambda x: x and identifier in x):
            element.decompose()
        for element in soup.find_all(id=lambda x: x and identifier in x):
            element.decompose()
    return soup.get_text()

2. Provenance-Aware Filtering

Calculate document quality scores differently if the source page contains forms or public submission architectures. Track metadata related to WordPress or Drupal footprints and apply a stricter quality classification threshold (e.g., downweighting or flagging content from pages with open, unauthenticated submission forms).

3. Temporal Consistency Audits

Verify content over multiple crawl epochs. If a page has historically remained static but suddenly receives a spike in text volume (typical of comment spam or automated injection), quarantine the newly added segments or fallback to older, cached versions of the page.

The Takeaway

Modern data curation pipelines operate almost entirely at the document level, leaving them fundamentally blind to localized, malicious injections. By treating user-generated comments identically to primary editorial content, LLM pretraining methods remain highly vulnerable to cheap, decentralized, and scalable computational propaganda campaigns.

Den's Take

For too long, pretraining safety papers have played in a sandbox, targeting highly curated, tiny domains like Wikipedia. The HALFLIFE framework injects some much-needed realism by analyzing what actually makes up the modern web: unvetted, messy public comment sections.

The fact that a cheap, automated comment campaign across 100k to 1M WordPress pages yields an estimated end-to-end inclusion probability of 0.13% in modern pretraining pipelines (like DCLM or FineWeb) is deeply concerning. It proves that our standard deduplication and heuristic filters are fundamentally ill-equipped to catch low-cost computational propaganda. While post-training alignment and fine-tuning defenses can help catch some risky behaviors, the sheer scale of raw web crawls means we have virtually zero visibility into what gets baked into base weights during pretraining.

However, my main reservation with this paper is the lack of empirical downstream evaluation on live web environments. While the authors successfully pretrained a model ladder (up to 1.3B parameters) on trace amounts of poison in a controlled setting, we do not have in-the-wild empirical verification on live websites due to ethical and safety limitations. Modeling survival through Common Crawl is a great proxy, but live web environments with active defenses could yield different inclusion dynamics.