Image generated by AI

TLDR

• What: CAPE (Compression-Aware Protective Evolution) is a proactive content-protection framework that injects human-invisible Unicode perturbations into high-value text to cause catastrophic semantic and structural degradation when processed by LLM-driven context compressors, without altering human readability.
• Who's at risk: Content owners, code repositories, and publishers whose proprietary assets are scraped and ingested by agentic workflows (e.g., LangGraph, GitHub Copilot, RAG pipelines).
• Key number: CAPE improves information degradation in compressed outputs by up to 75.8% compared to the strongest baseline while keeping human-visible text differences under 3.4% (with average difference as low as 1.4%).

Out of Sight: Weaponizing Context Compression to Block Agentic Web Crawlers

As large language model (LLM) agents like GPT-4.1, Gemini 3 Flash, and specialized coding tools like GitHub Copilot automate web navigation, traditional scraping barriers are failing. Perimeter defenses like robots.txt or Cloudflare's crawler blocking are easily bypassed by agentic crawlers that mimic human behavior or spoof legitimate browser sessions. This paper introduces CAPE, an active, content-layer defense that targets context compression—the critical pipeline chokepoint where agents condense massive scraped pages to fit tight prompt token limits—and corrupts the resulting outputs beyond recognition.

Threat Model

Background & Problem Setup

Traditional scraping defenses focus entirely on pre-delivery mechanisms. If an automated agent successfully fetches a page, the battle is usually lost. While prompt injections or jailbreaking techniques (e.g., Zou et al., 2023) can alter agent outputs, they often break human readability, fail to protect the core textual content, and raise compliance red flags.

Unlike these approaches, CAPE operates at the content layer. It relies on the insight that LLM agents must compress or summarize retrieved text to fit their context budgets (Jiang et al., 2023; Pan et al., 2024). By injecting optimized invisible characters (e.g., zero-width spaces, joiners, and control codes), CAPE disrupts the target LLM's tokenization and attention boundaries, inducing severe information loss.

Related Work Comparison

Methodology

CAPE optimizes invisible perturbations through a three-stage pipeline designed to scale from local white-box models to black-box commercial APIs:

[Raw Content]
      │
      ▼
┌─────────────────────────────────────────┐
│ 1. Structural Prior Discovery (Surrogate)│ ◄── Optimizes entropy & anomaly objectives
└─────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────┐
│ 2. Prior-Guided Evolutionary Adaptation  │ ◄── Discrete genetic search via structural priors
└─────────────────────────────────────────┘
      │
      ▼
┌─────────────────────────────────────────┐
│ 3. Preference-Calibrated Query Selection │ ◄── Local MLP ranker limits black-box queries
└─────────────────────────────────────────┘
      │
      ▼
[Protected Content (Invisible Perturbations Injected)]

Stage 1: Structural Prior Discovery

Because target compressors (e.g., GPT-4.1) are closed-source, CAPE first searches for vulnerabilities on an accessible surrogate model $M_s$ (such as Llama3-8B). It appends a fixed probe sequence $u_{1:T}$ and maximizes a multi-term objective to destabilize the model's token distribution:

$\arg \max_a \frac{1}{T} \sum_{t=1}^T \left[ H(q^a_t) + \lambda_a \log q^a_t(V_{\text{anom}}) - \lambda_l \log q^a_t(V_{\text{lang}}) \right]$

Where:

• $H(q^a_t)$ is the next-token entropy (encouraging generation uncertainty).
• $V_{\text{anom}}$ represents anomalous tokens (forcing the model to output junk/formatting characters).
• $V_{\text{lang}}$ represents standard, fluent text tokens (which are actively suppressed).

High-performing perturbations are parsed to extract structural priors—specifically local fragments (deleterious character combinations), fragment co-occurrences, and position-length attributes.

Stage 2: Prior-Guided Evolutionary Adaptation

Instead of performing a random search over individual characters, CAPE encodes candidate mutations into structural descriptors. Offspring inherit successful sequence configurations and mutation operators are guided by the structural priors mined in Stage 1. Candidates are filtered using a target-side fitness score:

$\tilde{F}_t(g) = D_t(g) + \lambda_p S_p(g) - \lambda_u U_t(g) - \lambda_r R_{\text{tabu}}(g)$

where $D_t(g)$ is the observed target-compressor degradation, $S_p(g)$ measures consistency with surrogate-derived priors, $U_t(g)$ penalizes unstable feedback, and $R_{\text{tabu}}(g)$ prevents the search from getting stuck in local minima.

Stage 3: Preference-Calibrated Query Selection

To prevent blowing through API limits, CAPE trains a lightweight local MLP ranker $f_\phi$ using pairwise logistic loss on historical query results:

$\mathcal{L}_{\text{align}}(\phi) = \sum_{(g_i, g_j) \in \mathcal{P}_t} \log \left( 1 + \exp(-y_{ij} \Delta^{ij}_\phi) \right)$

It uses an acquisition score to prioritize which candidates to evaluate on the target black-box model, balancing expected degradation, ranker uncertainty, and structural novelty.

Algorithmic Flow

# Conceptual algorithm of the CAPE framework
def apply_cape_protection(source_text, surrogate_model, target_api, query_budget=100):
    # Stage 1: Structural Prior Discovery
    raw_priors = surrogate_model.probe_and_optimize(source_text)
    seed_corpus = extract_structural_priors(raw_priors)
    population = initialize_population(seed_corpus)
    
    local_ranker = MLP_PreferenceRanker()
    query_history = []
    
    for round in range(max_rounds):
        # Stage 2: Prior-Guided Evolutionary Search
        candidates = generate_offspring_via_priors(population, seed_corpus)
        
        # Stage 3: Preference Calibration & Selection
        if len(query_history) >= min_pairs:
            local_ranker.train_on_pairwise_loss(query_history)
            
        # Allocate query budget using acquisition function (Eq 6/33)
        ranked_candidates = local_ranker.score_and_rank(candidates)
        queries_to_send = select_top_k(ranked_candidates, budget_limit=8)
        
        # Query target and record feedback
        for candidate in queries_to_send:
            deg_score = target_api.evaluate_compression(source_text, candidate)
            query_history.append((candidate, deg_score))
            
        population = update_population_via_annealing(candidates, query_history)
        
        if len(query_history) >= query_budget:
            break
            
    best_perturbation = select_best_overall(query_history)
    return inject_invisible_characters(source_text, best_perturbation)

Key Results

The authors evaluated CAPE across three domain tasks (long-form text, source code, and multi-turn dialogue histories) against commercial target models like GPT-4.1 and Gemini 3 Flash.

Metrics Definition

• Textual Degradation (TD): Measures surface corruption and fluency breakdown.
• Information Degradation (ID): Quantifies the loss of recoverable facts and semantics.
• Human-Visible Input Difference (HVID): Measures edits to the rendered text (lower is better).

Core Performance (Target: GPT-4.1)

Data source: Table 1, page 7.

Real-World System Transfers

1. LangGraph Agent Workflows (Table 2): CAPE caused a 59.7% Downstream Relative Accuracy Drop (DRAD) on task execution, while maintaining an HVID of only 2.8%.
2. GitHub Copilot (Table 3): CAPE achieved a 16.4% Completion Success Drop (CSD) and forced a 38.5% Malformed Output Rate (MOR), outputting unparseable code snippets, with just 3.4% HVID.

Critical Skepticism

While CAPE consistently outperforms baseline attacks under strict human-visibility constraints, source code is markedly more resistant to semantic degradation than natural language or conversational logs. Table 1 shows that CAPE achieves 42.8% TD on code compared to 49.2% on long-form prose. This is because programming syntax is structurally rigid; a model can often reconstruct missing context through localized dependency parsers even when formatting token structures are perturbed.

Limitations & Open Questions

• Vulnerability to Simple Preprocessing: As noted in the Limitations section, an adaptive attacker can neutralize CAPE entirely by implementing a basic regex sanitizer to strip out non-printable or zero-width Unicode characters. The authors argue that doing so risks stripping formatting-sensitive characters (e.g., in markdown or code indentation), but for standard web text, sanitization remains a simple, highly effective countermeasure.
• Context Window Expansion: As frontier models natively expand their context windows to extremely large budgets (e.g., millions of tokens), agent pipelines may bypass context compression steps entirely. If raw text is ingested directly without summarization, CAPE's primary disruption mechanism is bypassed.
• Tokenization Divergence: If the target LLM uses a vastly different tokenization strategy compared to the surrogate model used in Stage 1, the transferability of the structural priors drops, which increases the required query budget.

What Practitioners Should Do

For Content Publishers & Developers (Defenders)

1. Inject Non-Printable Spans: Programmatically inject zero-width spaces (\u200b), zero-width non-joiners (\u200c), or directional overrides into high-value text blocks before rendering them to the client.
2. Target Sensitive Regions: Focus injections near high-value boundaries. For natural language, inject near the end of paragraphs or adjacent to key named entities. For source code, focus on API declarations and class boundaries.

For LLM Engineers & Red Teamers (Scrapers/Consumers)

1. Sanitize Raw Web Scraping Payloads: If ingestion pipelines extract raw HTML or Markdown, enforce strict unicode sanitization to filter out non-printable zero-width markers:

import re

def sanitize_extracted_payload(html_content: str) -> str:
    # Remove zero-width spaces, joiners, and bidirectional overrides
    invisible_chars_pattern = re.compile(r'[\u200b-\u200d\ufeff\u200e\u200f]')
    return invisible_chars_pattern.sub('', html_content)

2. Leverage Structural Parsers: For code and structured data, rely on AST (Abstract Syntax Tree) parsers rather than pure text-based context compressors to identify and retain critical semantic code tokens.

The Takeaway

CAPE demonstrates that context compression is a highly viable defense layer against modern agentic crawlers. By moving content protection directly into the payload itself rather than relying on network-level blocks, CAPE allows publishers to protect their intellectual property without degrading the experience for human readers. However, as LLM ingestion pipelines integrate proactive sanitization tools, the future of content protection will require coordinated, multi-layered strategies spanning legal policies, platform metadata, and technical payload defenses.

Den's Take

This is a clever pivot in the cat-and-mouse game of web scraping. Instead of trying to block the initial connection—which is increasingly futile against automated agents—CAPE targets a structural pipeline bottleneck: tokenization and downstream context compression. Improving information degradation by up to 75.8% over the strongest baseline while keeping human-visible text differences under 3.4% via invisible Unicode characters (like zero-width spaces and joiners) is a highly practical, elegant result.

However, as a practitioner, I have to point out the fragility here. CAPE’s threat model relies heavily on the assumption that the agentic workflow must run some form of context compression. As context windows scale to millions of tokens, raw ingestion without summarization is becoming common, which would weaken this defense. Furthermore, any semi-competent scraper could neutralize these perturbations simply by sanitizing incoming text and stripping non-printable Unicode characters before tokenization.

Still, this work highlights the growing friction between content creators and AI platforms. This aligns with the broader realization that the unconstrained ingestion of retrieved web content introduces severe vulnerabilities into the LLM reasoning loop. CAPE proves the inverse: content owners can actively exploit this ingestion pipeline to defend their intellectual property, turning the agent's context window into a battleground.