Inference-Time Concept Suppression and Video-Centric Evaluation for Text-to-Video Models

Image generated by AI

TLDR

• What: SIRUS is a training-free, inference-time concept suppression framework for text-to-video (T2V) models that builds a contextualized text-embedding subspace to dynamically subtract concept residuals during sampling.
• Who's at risk: Deployments of modern open-source T2V generators (e.g., CogVideoX, Wan2.2, Open-Sora) used in public platforms where users might bypass prompt filters to generate unsafe, copyrighted, or style-cloned content.
• Key number: On CogVideoX, SIRUS achieves a 70.4% average forgetting success rate and lowers target concept frame hits to 25.7% (vs. 44.4% / 47.2% for VideoEraser) with an average video quality drop ($\Delta Q$) of only -0.016.

As text-to-video (T2V) models like CogVideoX, Wan2.2, and Open-Sora move toward production, the risk of unsafe or copyrighted generations has grown. Video generators present unique unlearning challenges due to temporal frame consistency and high sampling costs. Simple keyword filtering is easily bypassed, while weight-editing or fine-tuning is computationally expensive and hard to adapt to changing policies.

To address this, researchers proposed SIRUS (Subspace-Informed Residual Unlearning during Sampling), a training-free, inference-time framework that alters sampling trajectories to suppress target concepts while keeping underlying parameters frozen.

Threat Model

Background & Related Work

Concept unlearning has transitioned from image-level weight erasure to inference-time video interventions. Below is a comparison of current unlearning strategies:

Methodology: How SIRUS Erases Concepts on the Fly

SIRUS operates entirely at inference time with a frozen text encoder $E$ and denoising network $\epsilon_\theta$. The pipeline consists of four stages:

[Target Aliases] ---> (1) Concept Subspace Construction (SVD/QR) 
                                 |
                                 v
[Input Prompt]   ---> (2) Hierarchical Triggering & Local Projection ---> Edited Condition (c_t)
                                 |
                                 v
                     (3) Positive Concept-Reference Branch          ---> Reference Condition (c_c)
                                 |
                                 v
                     (4) Residual Unlearning during Sampling         ---> Output Video

1. Concept Subspace Construction

Rather than a static vector, SIRUS models concepts as a local linear subspace to account for linguistic context:

• Target aliases $A = \{a_1, \dots, a_m\}$ are placed in template prompts (e.g., "a video of [alias] in a scene").
• Prompts are encoded, and embeddings at the alias positions are extracted as samples $X \in \mathbb{R}^{N \times D}$.
• After centering, SVD, and QR orthonormalization, SIRUS produces a row-orthonormal basis $B = \{b_1, \dots, b_K\} \in \mathbb{R}^{K \times D}$.
• The projection operator onto this subspace is: $P_B(e) = \sum_{k=1}^K \langle e, b_k \rangle b_k$

2. Hierarchical Triggering and Prompt Projection

To prevent over-suppression on benign prompts, SIRUS uses an asymmetric trigger rule:

• Level 1: Exact alias/anchor matching.
• Level 2: Similarity search for tokens matching the concept subspace ($s_\ell = \max_k \langle e_\ell / \|e_\ell\|, b_k / \|b_k\| \rangle$).
• Levels 3 & 4: Conservative fallback steps for weak similarity.

Only triggered token embeddings are projected away from $B$ to form the edited condition $c_t$: $e'_\ell = e_\ell - \alpha_\pi P_B(e_\ell)$

3. Positive Concept-Reference Branch

To prevent the model from recovering concepts via pre-trained prior weights, SIRUS constructs a reference branch:

• Reference prompts are embedded and scored by similarity to the prompt anchor.
• The top-$k$ matches are aggregated via softmax to yield $R_c$.
• This is mixed with the unconditional embedding $c_u$ to produce $c_c$: $c_c = (1 - \gamma_\pi)c_u + \gamma_\pi R_c$

4. Residual Unlearning during Sampling

The model evaluates unconditional ($c_u$), edited ($c_t$), and reference ($c_c$) branches, outputting noise predictions $\epsilon_u$, $\epsilon_t$, and $\epsilon_c$. The standard classifier-free guidance (CFG) is: $\epsilon_{base} = \epsilon_u + s(\epsilon_t - \epsilon_u)$ The concept residual to be subtracted is defined as: $d_t = \epsilon_c - \epsilon_t$

SIRUS bounds the norm of $d_t$ to a profile budget $\rho_\pi\|q_t\|$ (where $q_t = \epsilon_t - \epsilon_u$), applies momentum smoothing to get $m_t$, and subtracts it during denoising via a cosine decay schedule: $\epsilon_{final} = \epsilon_{base} - \eta_\pi w_{i,\pi} m_t$

Key Results

SIRUS was evaluated on CogVideoX across five concepts: nudity (safety), church, garbage truck, parachute (objects), and Van Gogh (style).

1. Forgetting Success vs. Frame-Level Target Hit Rate

Table 1 reports the video-level unlearning success rate against the frame-level target hit rate (Success $\uparrow$ / Frame Hit $\downarrow$):

Analysis: Although the weight-updating T2VUnlearning gets high nudity unlearning success (88.0%), it does so by deleting the human subject entirely, dropping its person retention rate to 33.4% compared to SIRUS’s 78.5% (Table 7).

2. Video Quality Trade-Off (VBench Metrics)

Table 3 evaluates unlearned videos using VBench quality metrics (higher is better):

Limitations & Open Questions

1. Visually Persistent Objects remain Resistant: Large, visually dominant target concepts like "parachute" remain difficult (only 50.0% success rate) due to persistent visual features across frames.
2. Jailbreak Vulnerabilities: Under adversarial jailbreak prompts (Table 10), the failure rate of SIRUS rises to 24.0% (with a 21.9% frame target hit rate), though it still outperforms VideoEraser (78.0%).
3. Inference Latency: Running three forward passes during the erasure window increases single-video generation time by 23.8% (+43s on CogVideoX, Table 11).

What Practitioners Should Do

1. Deploy Prompt-Side Embedding Projection: Implement a localized embedding projection layer after the text encoder to project target-related tokens away from the concept's linear subspace:

   def project_embeddings(token_embeddings, concept_subspace_basis, alpha=1.0):
       # concept_subspace_basis: [K, D]
       projection = torch.matmul(token_embeddings, concept_subspace_basis.T)
       projection = torch.matmul(projection, concept_subspace_basis)
       return token_embeddings - alpha * projection
   

2. Integrate Multi-Branch CFG: Modify sampling loops to run the three-branch prediction (unconditional, edited, reference) only during the early sampling phase (determined by the cosine schedule), then revert to standard CFG to minimize latency.

3. Incorporate Object and Person Retention Auditing: When updating unlearning mechanisms, explicitly test with downstream object detectors (e.g., YOLO) to verify that target erasure does not trigger global scene collapse or subject disappearance.

Den's Take

I am generally skeptical of training-free safety interventions because they often trade away generation quality. However, SIRUS's approach of using SVD/QR to build a contextualized subspace and subtract concept residuals on the fly is highly elegant. Restricting average quality drop ($\Delta Q$) to just -0.016 while bringing average frame hits down to 25.7% is an impressive engineering feat for an update-free method.

That said, we must remember that this is an inference-time patch on frozen weights; the model still "knows" how to generate the unsafe content. As demonstrated in generative safety literature using tools like AutoDAN [22] or T2VSafetyBench [26], semantic adjustments and adversarial prompt optimization can bypass static text-space filters. If an attacker shifts the input distribution enough to slip past the constructed trigger subspace, the defense will fail because the underlying model weights remain unchanged. SIRUS is a great, cost-effective stopgap for deployment pipelines, but should be combined with output-side visual classifiers for comprehensive safety.