Image generated by AI

TLDR

• What: Activation-Guided GCG optimizes adversarial suffixes by minimizing projection onto a model's internal, low-dimensional refusal direction, paired with Soft-GCG (SGCG) which replaces discrete search with a continuous Gumbel-Softmax relaxation.
• Who's at risk: Open-weight LLMs (specifically Llama-2-7b-chat and Gemma 3 models below 12B parameters) deployed with white-box access or optimized locally before transfer attacks.
• Key number: Soft-GCG achieves a 33× to 43× speedup over standard GCG (2.5 minutes vs 81 minutes for 2,000 steps) while matching or exceeding standard attack success rates.

Cracking LLM Alignment from the Inside Out: Activation-Guided GCG and Gumbel-Softmax Suffix Optimization

Security researchers and red-teamers have long relied on Greedy Coordinate Gradient (GCG) to bypass LLM safety guardrails. However, standard GCG operates on a superficial behavioral proxy—optimizing inputs to force specific target output tokens like "Sure, here is..."—while ignoring the model's internal state. This paper by Cakar et al. (Harvard University) breaks the assumption that optimization must target output token logits. By fusing adversarial suffix generation with representation engineering, they demonstrate that targeting a model’s internal, low-dimensional "refusal direction" directly yields highly effective jailbreak suffixes. Simultaneously, they introduce a continuous relaxation using the Gumbel-Softmax distribution to solve the massive computational bottleneck of discrete suffix search.

Threat Model

Background & Related Work

This work sits at the intersection of discrete adversarial prompt optimization and representation-level safety interventions. Below is a comparison highlighting how this approach improves upon prior paradigms:

Technical Deep-Dive

1. Extracting the Target Refusal Direction

Following the difference-in-means methodology introduced by Arditi et al. (2024), the authors construct a dataset of paired harmful and harmless instructions. They record the residual stream activations at layer $\ell$ and token position $p$ (typically the last instruction token, $p = -1$) to compute the mean activation vector for both sets:

$\mathbf{r}^{(\ell)} = \mu_{\text{harmful}}^{(\ell)} - \mu_{\text{harmless}}^{(\ell)}$

This vector is normalized to construct the refusal direction unit vector:

$\mathbf{d}_{\text{refusal}} = \frac{\mathbf{r}^{(\ell)}}{\|\mathbf{r}^{(\ell)}\|}$

By analyzing candidate layers, the authors select the optimal layer $\ell^* = 14$ for Llama-2-7b-chat-hf as the target layer where the safety representation is most concentrated.

2. Activation-Guided Objective Functions

Rather than optimizing the standard cross-entropy loss over output tokens, Activation-Guided GCG replaces the objective with loss terms targeting the internal hidden states $\mathbf{h}_p^{(\ell)}(\mathbf{x})$ of the forward pass. They evaluate five spatial and depth hypotheses:

• Single: Targets only the single most critical activation layer and position: $\mathcal{L}_{\text{act}}^{\text{zero}}(\mathbf{x}) = \left(\frac{\mathbf{h}_p^{(\ell^*)}(\mathbf{x}) \cdot \mathbf{d}_{\text{refusal}}}{\|\mathbf{d}_{\text{refusal}}\|}\right)^2$
• Negative: Actively pushes the projection to be negative, driving the model to oppose refusal: $\mathcal{L}_{\text{act}}^{\text{neg}}(\mathbf{x}) = \frac{\mathbf{h}_p^{(\ell^*)}(\mathbf{x}) \cdot \mathbf{d}_{\text{refusal}}}{\|\mathbf{d}_{\text{refusal}}\|}$
• Layer: Probes spatial locality across all token positions at a fixed depth ($\mathcal{P}$ represents token positions): $\mathcal{L}_{\text{act}}^{\text{layer}}(\mathbf{x}) = \frac{1}{|\mathcal{P}|}\sum_{p \in \mathcal{P}} \left(\frac{\mathbf{h}_p^{(\ell^*)}(\mathbf{x}) \cdot \mathbf{d}_{\text{refusal}}}{\|\mathbf{d}_{\text{refusal}}\|}\right)^2$
• Token: Targets depth locality at a single position across all $L$ layers: $\mathcal{L}_{\text{act}}^{\text{token}}(\mathbf{x}) = \frac{1}{L}\sum_{\ell=1}^L \left(\frac{\mathbf{h}_p^{(\ell)}(\mathbf{x}) \cdot \mathbf{d}_{\text{refusal}}}{\|\mathbf{d}_{\text{refusal}}\|}\right)^2$
• All (Global): Suppresses the refusal projection globally across all layers and token positions: $\mathcal{L}_{\text{act}}^{\text{global}}(\mathbf{x}) = \frac{1}{L \cdot |\mathcal{P}|}\sum_{\ell=1}^L \sum_{p \in \mathcal{P}} \left(\frac{\mathbf{h}_p^{(\ell)}(\mathbf{x}) \cdot \mathbf{d}_{\text{refusal}}}{\|\mathbf{d}_{\text{refusal}}\|}\right)^2$

3. Soft-GCG: Bridging the Projection Gap

Discrete search in GCG is computationally expensive because it evaluates a large set of candidate substitutions ($K = 128$) via forward passes at every step. Soft-GCG replaces this with continuous optimization.

To maintain differentiability while ensuring the continuous representation can converge to discrete one-hot tokens, the authors parameterize the suffix using the Gumbel-Softmax distribution. At each suffix position $i$, they maintain a set of learnable logits $\phi_i \in \mathbb{R}^{|V|}$. During the forward pass, a "soft" token vector $\tilde{s}_i \in \Delta^{|V|-1}$ is calculated as:

$\tilde{s}_i = \text{Softmax}\left(\frac{\phi_i + g_i}{\tau}\right)$

where $g_i \sim \text{Gumbel}(0, 1)$ are i.i.d. samples used to escape local minima, and $\tau > 0$ is a temperature parameter. The input embedding for the suffix is then approximated by multiplying with the vocabulary embedding matrix $E \in \mathbb{R}^{|V| \times d}$:

$e(\tilde{s}_i) = \tilde{s}_i^\top E$

As temperature $\tau \to 0$, the soft token vectors converge to a discrete one-hot representation, minimizing the "projection gap" that typically breaks continuous prompt tuning methods like PEZ.

The authors use a 3-Phase "Slushy" Annealing Schedule to manage this transition:

1. Exploration (0% - 50% steps): $\tau$ decays from $2.5 \to 1.0$. High entropy allows broad traversal of the embedding space.
2. Refinement (50% - 75% steps): $\tau$ decays from $1.0 \to 0.5$. The structure begins to form while remaining flexible.
3. Solidification (75% - 100% steps): $\tau$ decays from $0.5 \to 0.01$, forcing the continuous soft vectors to snap directly to real discrete tokens.

Algorithmic Implementation

import torch
import torch.nn.functional as F

def soft_gcg_step(model, embeddings_matrix, phi, prompt_tokens, target_tokens, temp, lr=0.1):
    """
    Performs a single step of Soft-GCG optimization.
    phi: Learnable suffix logits [suffix_len, vocab_size]
    """
    optimizer = torch.optim.Adam([phi], lr=lr)
    optimizer.zero_grad()
    
    # 1. Sample from Gumbel noise
    gumbel_noise = -torch.log(-torch.log(torch.rand_like(phi) + 1e-20) + 1e-20)
    
    # 2. Compute soft token probabilities
    soft_tokens = F.softmax((phi + gumbel_noise) / temp, dim=-1)
    
    # 3. Project to continuous embedding space
    soft_embeddings = torch.matmul(soft_tokens, embeddings_matrix) # [suffix_len, embed_dim]
    
    # 4. Concatenate prompt, soft suffix, and target embeddings
    prompt_embeds = model.embed_tokens(prompt_tokens)
    target_embeds = model.embed_tokens(target_tokens)
    input_embeds = torch.cat([prompt_embeds, soft_embeddings, target_embeds], dim=0)
    
    # 5. Forward pass and compute loss (e.g., Carlini-Wagner Loss)
    logits = model(inputs_embeds=input_embeds.unsqueeze(0)).logits
    loss = compute_carlini_wagner_loss(logits, target_tokens)
    
    # 6. Backward pass to update logits
    loss.backward()
    optimizer.step()
    
    return loss.item()

Experimental Results

The authors systematically evaluated their approaches on Llama-2-7b-chat-hf using the AdvBench dataset and compared them against standard baseline runs.

1. Activation-Guided GCG vs. Baselines (200 Steps)

Evaluating different activation loss objectives against standard GCG reveals a massive boost in sample efficiency when targeting internal activations:

Critical Observations on Table 1:

• The Good: At 200 steps, the "All" (Global) objective achieves a 0.91 Substring ASR—outperforming standard GCG (0.76) and approaching the ablation limit of 0.98. This demonstrates that suppressing the refusal representation globally yields highly effective, sample-efficient gradients.
• The Catch: Despite the high Substring ASR (which checks for simple evasion of refusal phrases like "I cannot"), the actual HarmBench ASR for the "All" objective is 0.01 (1%), and LlamaGuard 4 is only 0.09. This shows that while the attack successfully suppresses the generation of direct refusal phrases, it often fails to steer the model into generating high-quality harmful payloads. The Single layer/position loss actually achieved a slightly higher HarmBench ASR (0.04), but still remains practically ineffective under robust evaluations.

2. Continuous Soft-GCG vs. Vanilla GCG Efficiency

The real breakthrough of the paper is the optimization speed. By bypassing discrete combinatorial search, Soft-GCG (SGCG) achieves parity in attack success while running vastly faster:

• Vanilla GCG (2000 steps): 81 minutes
• Soft-GCG (2000 steps): 2.5 minutes
• Speedup: 33× (with shorter 500-step runs reaching ~43× speedup).

3. Scaling Resistance in Gemma 3

To test how these vulnerabilities hold up to scale, the authors ran SGCG against the Gemma 3 family. The results demonstrate a brutal scaling cliff:

ASR marked with an asterisk for the 270m model due to output incoherence post-optimization.

This scaling behavior suggests a representational phase transition: larger models do not represent safety as a simple, low-dimensional linear subspace that can be easily suppressed via gradient optimization. Instead, safety constraints appear to be encoded in higher-rank, non-linear manifolds.

Limitations & Open Questions

1. The Scaling Cliff: SGCG completely fails (0.00% ASR) on the 12B parameter Gemma 3 model. If scaling naturally distributes refusal representations beyond the reach of low-dimensional projections, representation-guided attacks may be fundamentally limited to smaller model families.
2. Incoherent Suffixes: On ultra-small models (like Gemma3-270m), successful attacks often degrade the model's language capabilities entirely, leading to incoherent generations rather than useful jailbreaks.
3. The Substring Metric Illusion: High Substring ASR does not equal a high-quality jailbreak. Suppressing the refusal direction stops the model from saying "I'm sorry", but it doesn't guarantee the model will successfully complete the harmful prompt. Often, the model ends up in a state of confused babble.

What Practitioners Should Do

If you are defending, auditing, or deploying LLMs, you should adapt your security posture immediately:

1. Abandon Shallow Evaluation Metrics: Do not measure safety robustness using substring matching (looking for "I'm sorry" or "As an AI"). This paper proves that activation-guided attacks can easily silence these behavioral indicators while failing to secure or fully break the model. Use multi-turn, LLM-as-a-judge evaluators like LlamaGuard or HarmBench.
2. Harden Internal Representations via Adversarial Training: Standard alignment is a surface-level behavioral patch. Model developers should employ adversarial training that penalizes models when their internal activations map to low-dimensional steering vectors (like known refusal directions).
3. Inference-Time Activation Monitoring: Implement lightweight projection checks on the residual streams at key bottlenecks (e.g., intermediate layers like layer 14 on 7B models). If the projection of the activation onto the known refusal direction is anomalously low while handling sensitive topics, flag the generation.
4. Adopt Hybrid Defenses: Combine weight-orthogonalization techniques (which project out refusal steering directions) with external input guardrails to prevent attackers from finding continuous trajectories into the model's soft embedding spaces.

The Takeaway

Activation-Guided GCG and Soft-GCG prove that behavioral alignment in small models is remarkably thin; their internal safety mechanisms are low-dimensional and incredibly vulnerable to continuous optimization. However, the complete failure of these attacks on models above 12B parameters points to a structural shift in how larger models organize safety representations, highlighting that raw model scale remains one of our best structural defenses.

Den's Take

Standard GCG has always felt like a brute-force sledgehammer—deeply impractical for real-world red teaming due to its massive computational overhead. That’s why this work excites me: Cakar et al. actually make suffix optimization practical by replacing discrete searches with Gumbel-Softmax relaxation (Soft-GCG), bringing optimization times for 2,000 steps down from 81 minutes to just 2.5 minutes.

But the real intellectual leap here is shifting the optimization target from superficial behavioral token matching (like forcing the model to output "Sure, here is") to targeting the internal residual stream's refusal direction. As I have argued previously, shallow behavioral alignment is easily bypassed because it fails to secure deeper representation layers; this paper proves that point mathematically by showing that directly minimizing activation projection onto this low-dimensional refusal vector completely bypasses safety guardrails.

However, let's keep our boots on the ground. The white-box requirement during the optimization phase means this is not an instant exploit against closed-weights APIs, even if the resulting suffixes do exhibit some transferability. Still, for local security auditing of open-weight models like Llama-2-7b-chat and Gemma 3, this drastically lowers the bar for generating highly effective exploits.