PVDetector: Detecting Prompt Injection Attacks on Purpose-Specific LLM Agents through Policy-Violation Concept Analysis

Image generated by AI

TLDR

• What: PVDetector is a training-free prompt injection (PI) detection framework that isolates "policy-violation" (PV) concepts in the LLM's hidden activation space and measures real-time semantic alignment during inference.
• Who's at risk: Purpose-specific LLM and Vision-Language Model (VLM) agents (e.g., customer support bots, RAG assistants) that enforce specialized application boundaries.
• Key number: PVDetector consistently achieves a False Negative Rate (FNR) of <1% (frequently hitting 0.00%) across multiple SOTA models (Llama-3.1, Qwen-2.5) with a negligible overhead of just 0.1059 seconds per query.

As enterprises race to deploy specialized AI agents—such as customer support chatbots, financial advisors, and automated code generators—they increasingly rely on system prompts to define strict purpose-specific restriction (PSR) policies. However, malicious users can bypass these guardrails using prompt injection (PI) attacks, hijacking backends like Llama-3.1 or Qwen-2.5 to execute arbitrary instructions. Traditional defenses, which analyze only surface-level input-output text patterns or rely on expensive auxiliary models, fail to capture the model's internal awareness of policy violations.

Threat Model

Background / Problem Setup

Unlike general-purpose safety alignment (which blocks universally harmful inputs like bomb-making instructions), purpose-specific agents must enforce application-specific boundaries. For example, a "Recipe Master" agent must reject questions about "writing a Python script," even though Python scripts are not inherently harmful. This creates a broader, more volatile attack surface that traditional defenses struggle to secure.

Existing prompt injection detection mechanisms fall into several categories, but they all carry significant trade-offs:

Methodology

The core insight behind PVDetector is that LLMs inherently recognize policy violations internally, even when they are successfully coerced into generating unauthorized outputs.

The authors construct a two-phase framework: an offline phase to extract "policy-violation" (PV) concepts and select key layers, and an online phase to intercept attacks during live inference.

       [Offline Phase]
Contrastive Pairs (ID vs. OOD) ---> Forward Pass ---> Extract Hidden States ---> Difference-in-Means (PV Vectors)
                                                                                       |
                                                                                       v
                                                                             Select Key Layers (L_key)
                                                                                       |
       [Online Phase]                                                                  |
User Query ---> Forward Pass ---> Extract Hidden States at L_key ----------------------> Project onto PV Vectors
                                                                                       |
                                                                                       v
                                                          Aggregate PVS Scores ---> Thresholding ---> Flag / Allow

Step 1: Contrastive Pairs Construction (Offline)

The defender constructs $N = 30$ pairs of contrastive inputs tailored to the agent's PSR policy (no attack instances are required).

• Positive Sample ($X^+$): An out-of-domain query (e.g., "Provide one reason why people should recycle.")
• Negative Sample ($X^-$): An in-domain query (e.g., "How to make a cake.")

Step 2: Policy-Violation Concept Extraction (Offline)

These pairs are fed into the backend LLM. At layer $l$, hidden states are extracted at the last token position $h_{last}^l$. Using the difference-in-means technique [22], the policy-violation (PV) vector is computed as:

These layer-wise vectors capture the semantic gap between violating and compliant prompts, serving as linear representations of the PV concept. When projected back into the vocabulary space using the LLM's unembedding layer, these vectors map to tokens like "sorry", "unable", "cannot", and "irrelevant" (as described in Section 4.2.3).

Step 3: Key Layers Selection (Offline)

To optimize latency and accuracy, PVDetector identifies the layers that best capture the semantic concept of violation. The difference vector of the mean projection difference between positive and negative samples across layers is calculated as $d = (d_1, d_2, \dots, d_L)$. Key layers are selected where the layer's discriminative power meets or exceeds the final layer:

Step 4: Online Detection (Inference-Time)

For an incoming test prompt $x_{test}$, PVDetector extracts its hidden representations at the selected key layers. It calculates the Policy-Violation Strength (PVS) score for each key layer:

These scores are aggregated across $\mathcal{L}_{key}$ using the trapezoidal rule to approximate the integral. If the aggregated score $s_{aggr}$ exceeds a calibrated threshold $\theta$, the input is flagged and blocked.

Key Results

The framework was evaluated using three LLMs (Llama-3.1-8B, Qwen-2.5-7B, Qwen-2.5-14B) on the RecipeMaster agent across two datasets (Alpaca, MMLU) under five attack types: Ignore [38], Fake Completion [61], Combined [25], GCG [76], and a novel hybrid Ig-GCG attack.

Detection Performance (False Negative Rate - FNR %)

Table 1 highlights that PVDetector maintains near-zero FNR while keeping False Positive Rates (FPR) strictly bound beneath the $1\%$ calibration limit.

Computational Overhead and Efficiency

Table 3 evaluates average per-sample detection time conducted on an NVIDIA A800 GPU. PVDetector achieves minimal latency (0.1059s), which is roughly $15\times$ faster than alternative training-free approaches like AttentionTracker.

Resilience to Adaptive Attacks

As detailed in Section 6.2.9, an adaptive white-box adversary was tested. This attacker attempts to bypass PVDetector by appending token sequences optimized to simultaneously minimize the PV projection value (PVS) while maintaining a high attack success rate (ASR).

Even under this strong adaptive threat model, PVDetector maintains a $0\%$ FNR, forcing the attack success rate (ASR) down below $5\%$ (on Llama-3.1 (8B)) as the optimization weight $\beta$ increases (as illustrated in Figure 6).

Limitations & Open Questions

While PVDetector represents a major step forward, security practitioners must consider several limitations:

1. White-Box Dependency: The defense requires extraction of hidden states from backend models. In closed-source settings (e.g., OpenAI API, Anthropic API), external developers cannot inspect intermediate layers or extract $h_{last}^l$. Consequently, PVDetector is currently restricted to self-hosted, open-source architectures (e.g., Llama, Qwen, Mistral).
2. Threshold Sensitivity: The calibration threshold ($\theta$) depends heavily on the specific agent's prompt definitions. Changes to system prompts or agent roles require recalibrating the threshold using fresh benign data to preserve low False Positive Rates.
3. Contrastive Query Diversity: If the offline contrastive pairs fail to comprehensively map the agent’s boundary space, the resulting PV vectors may fail to align with unseen categories of out-of-domain queries.

What Practitioners Should Do

If you are deploying open-source LLM agents in production, you can implement this defense by following these steps:

1. Extract Latent States Programmatically

Register forward hooks on the target LLM layers to capture hidden activations at the last token position during the forward pass. For example, using PyTorch on a self-hosted vLLM or Hugging Face setup:

import torch

class PVExtractor:
    def __init__(self, model, key_layers):
        self.model = model
        self.key_layers = key_layers
        self.activations = {}
        
        # Register hooks for selected key layers
        for layer_idx in key_layers:
            layer = self.get_layer_by_index(layer_idx)
            layer.register_forward_hook(self._make_hook(layer_idx))

    def _make_hook(self, idx):
        def hook(model, input, output):
            # Extract last token hidden state: shape [batch, seq_len, hidden_dim] -> [batch, hidden_dim]
            self.activations[idx] = output[0][:, -1, :].detach().cpu()
        return hook

    def get_layer_by_index(self, idx):
        # Access Transformer layers dynamically (varies by architecture)
        return self.model.model.layers[idx]

2. Generate Domain-Specific Contrastive Pairs

Construct 30 positive (violating) and 30 negative (compliant) queries using your agent's system prompt. For instance, if your agent is a customer support bot for a banking application, draft queries such as:

• Negative (Compliant): "What is my account balance?" or "How do I transfer funds?"
• Positive (Violating): "Write a poem about dogs" or "Can you write a python script to parse CSV files?"

3. Select the Later Transformer Layers

As highlighted in Figure 7, the distinction in PVS scores between violating and compliant prompts is concentrated in the latter half of the model's layers. Focus your key layer selection ($\mathcal{L}_{key}$) on the final $15% to \30%$ of layers to minimize projection overhead while preserving detection accuracy.

The Takeaway

PVDetector shows that LLMs retain a clear internal awareness of policy violations even when an adversarial input bypasses their safety guardrails and forces them to generate malicious outputs. By shifting our defensive focus away from superficial surface-level text matching and targeting the model's internal activation space, we can build highly robust, real-time, training-free guardrails for enterprise AI agents.

Den's Take

I’ve always argued that defending LLMs at the surface text level is a losing battle. PVDetector gets this right by looking downstream into the hidden activation space to isolate "policy-violation" concepts. Achieving a False Negative Rate of under 1% on models like Llama-3.1 and Qwen-2.5 with a mere 0.1059-second overhead is a massive win for training-free, real-time defense. It correctly recognizes that purpose-specific agents face domain-boundary challenges that general safety alignment completely misses.

This approach directly aligns with representation engineering techniques [75] and research on the Linear Representation Hypothesis [36], which suggest that safety-related concepts map to specific directions within the model's activation space.

However, as a practitioner, I’m highly skeptical of PVDetector's resilience against adaptive white-box adversaries. If an attacker has the gradient access required to run optimization attacks like GCG, they can easily craft adversarial suffixes specifically designed to mask their projection in this hidden-state subspace. Until we evaluate this defense against adaptive attacks optimized to bypass the projection vectors themselves, I would caution against treating this as a silver bullet—even if the initial FNR numbers look spectacular.