DataShield: Uncovering Risky Fine-Tuning Data Across LLMs Through Consensus Subspace Alignment

Image generated by AI

TLDR

• What: DataShield is a defense framework that prevents safety degradation during LLM fine-tuning by extracting consensus safe/unsafe semantic subspaces from an ensemble of reference models to filter or mask risky data, without requiring target model access.
• Who's at risk: Enterprise teams fine-tuning open-weight instruction models (e.g., Phi3, Qwen3, Gemma2/3) on custom datasets, which can silently compromise their safety guardrails and make them vulnerable to malicious exploits.
• Key number: DataShield-Sm segment masking reduces HarmBench Attack Success Rate (ASR) on Phi3-medium-4k-it from 76.0% down to 17.1%, while preserving general downstream utility.

Fine-tuning is the standard paradigm for adapting LLMs to domain-specific tasks. However, recent AI security research reveals a silent and severe vulnerability: supervised fine-tuning (SFT) on seemingly benign task data can systematically degrade safety alignment, rendering guardrails in target models entirely ineffective against malicious jailbreaks. Traditional data-filtering defenses are highly model-dependent, relying on target-specific gradients, tokenizers, or representations, which makes them expensive, hard to scale, and unusable in black-box scenarios.

To resolve these transferability limitations, researchers have introduced DataShield, a data assessment framework that decouples the data-filtering step from the target model. By extracting consensus safe and unsafe representation subspaces from a pool of independent safety-aligned source LLMs, DataShield sanitizes downstream datasets before the target model ever sees them.

Threat Model

Background & Problem Setup

The core issue with existing safety-preserving data filtering is model dependency. A data sample that looks benign to one model's internal representation might trigger safety degradation in another model during fine-tuning. The table below compares DataShield to prior defensive strategies cited in the paper.

Methodology

DataShield identifies risky fine-tuning data through a multi-stage consensus subspace alignment pipeline.

# DataShield Preprocessing Pipeline (Abstracted)
# Stage 1: Build Consensus Subspaces
subspaces = {}
for model in source_models:
    layers = find_safety_layers(model, over_refusal_data)
    H_safe = extract_representations(model, layers, safe_references)
    H_unsafe = extract_representations(model, layers, unsafe_references)
    
    # SVD to get top-d eigenvectors
    U_safe = spectral_decomposition(H_safe, d=16)
    U_unsafe = spectral_decomposition(H_unsafe, d=16)
    subspaces[model] = (U_safe, U_unsafe)

# Stage 2: Risk Scoring
if mode == "sample_filtering": # DataShield-Sp
    for sample in downstream_dataset:
        score = compute_average_alignment_gap(sample, subspaces)
    filtered_dataset = remove_top_fraction(downstream_dataset, score, rho=0.20)
    
elif mode == "segment_masking": # DataShield-Sm
    for sample in downstream_dataset:
        segments = tokenize_independent_split(sample.response)
        for segment in segments:
            # Compute decoupled incremental alignment gap
            segment.score = compute_decoupled_risk(segment, subspaces)
    masked_dataset = mask_top_fraction_segments(downstream_dataset, segments, rho=0.20)

Step 1: Identifying Safety-Critical Latent Spaces

Instead of using all hidden layers, DataShield uses weight-perturbation analysis (following Li et al., 2025a,c) on a benign over-refusal dataset ($D_{or}$, containing 110 instructions) to locate the top $N_{crit} = 3$ layers of each source model $M_{src}^{(k)}$ that are most sensitive to safety alignment behavior.

Step 2: Extracting Consensus Subspaces

For each selected layer, representations are extracted using 100 safe and 100 unsafe reference responses from the reference dataset (as utilized in Zou et al., 2024; Li et al., 2025a). Rather than computing a simple mean vector—which limits the safety semantic representation—DataShield applies semantic spectral decomposition to construct safe and unsafe orthonormal bases, $U_{safe}^{(k)}$ and $U_{unsafe}^{(k)}$, retaining the top $d = 16$ principal directions.

Step 3: Risk Estimation via Subspace Alignment

The alignment metric $\phi(\tilde{h}, U)$ measures the projection of a hidden state $\tilde{h}$ onto a subspace $U$:

• Sample-Level Filtering (DataShield-Sp): Computes the difference between alignment with $U_{unsafe}$ and $U_{safe}$. Samples with scores in the top $\rho = 0.20$ intervention budget are removed.
• Segment-Level Masking (DataShield-Sm): Splits responses into tokenizer-independent character spans (sentences/lines). To prevent preceding context from leaking into subsequent safe tokens, it implements Autoregressive Risk Decoupling:

Segments containing high decoupled risk scores are masked from the supervised fine-tuning loss.

Key Results

To test transferability, DataShield's consensus subspaces were constructed using Llama3-8B-Instruct, Qwen2.5-7B-Instruct, and Mistral-7B-Instruct-v0.3. Pre-filtered datasets were then used to fine-tune four completely unseen target models.

Table 1 evaluates the transfer performance of these filters on the Alpaca dataset across different architectures:

Analysis of the Results:

1. The Token-Masking Baseline Collapse: Prior token-level masking (TOSS) collapses during cross-architecture transfer. For example, on Gemma2-27B-it, fine-tuning with TOSS yields a HarmBench ASR of 52.1%, which is actually worse than Standard SFT's 51.3%. This is because TOSS relies directly on reference-model tokenizer token boundaries, which do not translate to target architectures. DataShield-Sm bypasses this with tokenizer-independent lexical segmentation, preserving safety much more effectively.
2. Computational Overhead Savings: As shown in Table 3 of the paper, DataShield is highly efficient. When preprocessing about 14K Dolly examples on Gemma2-27B-it, DataShield-Sp requires only 47.9 GB of peak memory and 64 minutes, whereas SEAL requires 291 GB and 835 minutes—a ~13x execution speedup and ~6x memory footprint reduction.

Limitations & Open Questions

• Ensemble Blind Spots: DataShield constructs consensus spaces using a specific set of source models. If the selected source models share systemic safety blind spots, the consensus alignment subspace will mirror those gaps.
• Slight Utility Trade-off: Although DataShield maintains performance close to the SFT baseline, it still incurs a consistent general utility drop of ~1% (e.g., SLIMORCA score dropping from 68.2% to 67.3% on Phi3). For production environments where downstream performance is critical, tuning the budget $\rho$ is required.
• Generalization to Ultra-Large Models: While evaluations demonstrate successful transfer up to large target models (e.g., Qwen2.5-72B-Instruct and Llama3-70B-Instruct as shown in Table E.6), future work is needed to explore scaling to even larger frontier models or complex mixture-of-experts architectures.

What Practitioners Should Do

1. Move Away from Single-Model Target Filters: If you are hosting a model-fine-tuning pipeline, do not rely on safety signals from a single model. Use an offline reference ensemble consisting of at least three diverse model families (e.g., Llama 3, Mistral, and Qwen 2.5) to build a combined consensus space.
2. Implement Tokenizer-Agnostic Lexical Segmentation: When sanitizing fine-tuning datasets, tag spans using sentence boundary detection tools (such as NLTK or spaCy's default English tokenization settings) rather than using token indices. Map these character spans to target-model tokenizers dynamically during SFT.
3. Use Autoregressive Risk Decoupling: If you choose to mask risky response segments, do not use absolute risk values. Use incremental differences in risk scores ($\Delta q_t = \frac{1}{2}(q_t - q_{t-1})$) to localize safety violations. This prevents the carry-over of safety flags to subsequent benign segments, preventing unnecessary over-masking.
4. Tune the Intervention Budget ($\rho$) dynamically: Sweeping over $\rho \in \{0.1, 0.4, 0.6, 0.8\}$ using an in-house evaluation dataset (e.g., GSM8K or HumanEval) is recommended to find the best compromise between downstream performance and safety.

The Takeaway

Fine-tuning safety degradation is a silent vulnerability that undermines LLM guardrails. DataShield demonstrates that safety representations are highly transferable across diverse models when modeled as consensus subspaces rather than rigid, model-specific vectors. Moving forward, data curation for LLM adaptation must shift from target-specific filtering to tokenizer-independent, semantic-level sanitization pipelines.

Den's Take

I am highly encouraged by DataShield's focus on target-independent, black-box safety filtering. In the real world, security practitioners rarely have the luxury of accessing target-model gradients or internal layers during rapid downstream deployment. Moving the defense entirely upstream by leveraging an offline ensemble (Llama3-8B, Qwen2.5-7B, and Mistral-7B) to construct consensus safe and unsafe subspaces is a highly pragmatic design pattern.

The empirical results are genuinely impressive: dropping the HarmBench Attack Success Rate (ASR) on Phi3-medium-4k-it from a vulnerable 76.0% down to 17.1% via segment masking (DataShield-Sm) proves that token-level intervention can prevent alignment degradation while preserving general utility. Mapping safe and unsafe subspaces is a robust and scalable approach compared to relying on target-specific, low-level model states.

However, we must not overlook the core limitation of transferability. This consensus-based approach assumes that the target model's internal representation space correlates sufficiently with the chosen reference ensemble. If you are fine-tuning a model with a radical architecture or a highly divergent alignment training recipe, the "consensus" subspace may suffer from blind spots, potentially letting subtle training exploits slip through the cracks undetected.