Minimal Prompt Perturbations Lead to Code Vulnerabilities: Prompt Fragility and Hidden-State Signals in Coding LLMs

Image generated by AI

TLDR

• What: Minimal, benign prompt mutations (like single-character typos) can silently flip LLM-generated code from secure to vulnerable, but these security vulnerabilities can be predicted directly from the model's prompt-end hidden states before code generation.
• Who's at risk: Enterprise developer workflows utilizing LLM-based coding assistants (e.g., Cursor, GitHub Copilot, Amazon Q) that lack strict post-generation verification or input sanitization.
• Key number: Input-handling vulnerabilities are highly predictable from prompt hidden states with a mean Area Under the ROC Curve (AUC) of 0.753 ± 0.038, whereas secure-default vulnerabilities are much harder to detect, yielding a mean AUC of only 0.674 ± 0.037.

LLM-based coding assistants like GitHub Copilot, Cursor, and Amazon Q have integrated deeply into the modern software development lifecycle. While developer productivity has soared, the security posture of the generated code remains highly precarious. Although developers routinely paraphrase prompts, fix minor typos, or rename variables without expecting changes in the output, recent research by Sternfeld et al. (arXiv 2026) reveals that coding LLMs suffer from severe prompt fragility—where a single-character mutation can silently strip security logic, rendering the resulting code exploitable.

Threat Model

Background & Problem Setup

Previous studies on prompt robustness focused almost exclusively on functional correctness—measuring whether minor edits caused code to fail compile-time or run-time tests. This left a massive security blind spot. Sternfeld et al. (arXiv 2026) closed this gap by evaluating prompt fragility using the CWEval benchmark (Peng et al., 2025), which uniquely couples functional tests with security-specific test oracles.

The comparison below highlights how this study builds upon and differs from related security evaluation and robustness work:

Unlike PoisonedRAG (Zou et al., CCS 2024), which demonstrates adversarial manipulation of retrieval corpora, Sternfeld et al. (arXiv 2026) show that vulnerability generation does not require a malicious adversary—ordinary developer typos are enough to trigger exploitable bugs.

Methodology

The authors systematically mutated prompts across three categories:

1. Single-character changes: Replacing one character in the prompt with a random ASCII letter.
2. Three-character changes: Swapping three characters in a single token with random ASCII letters.
3. Whole-token replacements: Substituting a token with one of its top-10 nearest neighbors based on tokenizer embedding cosine similarity.

These mutated prompts were run against CodeLlama (70B), DeepSeek-Coder (33B), and Qwen3-Coder (30B) at temperature $T = 0$ across five programming languages (C, C++, Go, JavaScript, and Python).

To determine whether these vulnerabilities are encoded in the model's prompt representations before generation even begins, the authors trained lightweight linear (L2-regularized logistic regression) and non-linear (2-layer Multi-Layer Perceptron) classifiers on the hidden states of the prompt's final token. The conceptual pipeline for this probing architecture is illustrated below:

import torch
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

# 1. Extract prompt's last-token hidden states from the optimal transformer layer
# Shape: [num_prompts, hidden_dimension]
X_train, y_train = load_hidden_states_and_security_labels(split='train')
X_test, y_test = load_hidden_states_and_security_labels(split='test')

# 2. Train a linear probe to predict final code security prior to token generation
probe = LogisticRegression(penalty='l2', C=0.5)
probe.fit(X_train, y_train)

# 3. Output prediction probability of the code being jointly functional and secure
preds = probe.predict_proba(X_test)[:, 1]
print(f"Probe Held-Out AUC: {roc_auc_score(y_test, preds):.3f}")

Key Results

1. Prompt Fragility in Action

Figure 5 displays a striking real-world case where a single-character mutation in a spec prompt flips the output from secure to vulnerable. When prompting DeepSeek-Coder-33B for CWE-022 (Path Traversal in Python), mutating the word otherwise, to otherwiseV (token 26/102, a ~25% depth change) completely stripped the path-traversal check (is_within function logic) from the final generated code, silently allowing malicious archive extraction outside the destination folder.

2. Baseline Model Security Performance

Table 2 in the paper outlines the baseline (non-mutated) pass rates of these models on CWEval, assessing both functional correctness (func) and joint functional and secure output (func-sec):

As Figure 2 illustrates, Qwen3-Coder (30B) is the most robust of the three models, exhibiting consistently smaller degradation bars when subjected to mutations.

3. Probing the Hidden States

When evaluating the probing classifiers on the held-out test set, Sternfeld et al. (arXiv 2026) discovered that the predictability of code security varies dramatically by vulnerability type. They classified the 18 CWEs into two distinct groups, as shown in Table 3:

A one-sided Mann-Whitney U test on the per-CWE AUC values yielded $U = 68$ with a $p$-value of 0.009, confirming a statistically significant gap in predictability between the two groups.

Why is there a predictive gap?

As Section 4.4 describes, input-handling vulnerabilities require a structural fix (such as inserting a validation block or a sanitization function). The LLM must commit to this structural change early in the generation process, which is heavily reflected in the prompt-end hidden states. Conversely, secure-defaults flaws hinge on localized keyword choices (e.g., choosing a strong cipher or setting a safe flag). These decisions are deferred to next-token sampling during decoding, meaning they leave almost no signature in the initial prompt representation.

Limitations & Open Questions

• Closed-Source Black Box Probing: Probing is fundamentally limited to open-weight architectures. Security researchers cannot extract hidden states from proprietary APIs like OpenAI's GPT-4o or Anthropic's Claude 3.5 Sonnet, limiting real-time pre-generation detection on those platforms.
• Dynamic Probing: Probing was restricted to the static, prompt-end representation. Extending probes to dynamic hidden states during generation might capture the deferred choices characterizing Group D (secure-defaults) vulnerabilities.
• Manual CWE Partitioning: The division of CWEs into Input Handling (Group I) and Secure Defaults (Group D) was conducted manually based on canonical fixes, meaning a more generalized, automated taxonomy remains an open challenge.

What Practitioners Should Do

1. Implement Pre-Generation Prompt Normalizers: Since single-character typos drastically shift security logic (e.g., the otherwise, $\rightarrow$ otherwiseV path-traversal flip), deploy automatic spell-checkers and grammar normalizers (such as LanguageTool or native LLM-based sanitizers) in developer environments before submitting prompts to coding LLMs.
2. Deploy Hidden-State Probes for Open-Weight Copilots: If hosting an in-house instance of a coding model like Qwen2.5-Coder or DeepSeek-Coder, train lightweight probing classifiers on the prompt-end representations. Set a threshold (e.g., AUC > 0.75) to intercept generations predicted to omit input-handling guards.
3. Rely Heavily on Post-Generation SAST for Secure Defaults: Because secure-defaults issues (like weak cipher configurations) escape prompt-level probing (AUC of 0.674), configure your CI/CD pipelines with rigid Static Application Security Testing (SAST) tools like Semgrep or Bandit to catch low-level, next-token security slips.
4. Enforce $T=0$ for Generation Stability: Keep the model's generation temperature set strictly to 0.0. While sensitivity analyses in Section A.3 show that higher temperature ($T=0.8$) does not alleviate prompt fragility, higher temperatures introduce non-deterministic security behavior.

The Takeaway

The generative code pipeline is incredibly fragile: minor, benign edits to developers' prompts can silently deactivate robust security guards. By proving that a model's internal prompt representation predicts these vulnerability flips before a single line of code is written, Sternfeld et al. (arXiv 2026) open a promising new path for real-time, zero-latency security guardrails in enterprise AI coding assistants.

Den's Take

The fact that a single-character typo can silently flip generated code from secure to highly vulnerable isn't just a quirky academic edge case—it is a massive liability for any enterprise relying on automated software development. Imagine a developer pushing a rushed hotfix to a $15M payment gateway pipeline via GitHub Copilot, only for a minor keyboard slip to silently strip out input-sanitization logic. This is how the next devastating software supply chain breach will happen.

What excites me about this work, however, is the defense vector: utilizing prompt-end hidden states to predict these "silent flips" before token generation even begins. This aligns directly with my prior analysis in Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals, where I argued that robust LLM security cannot rely on static input checks and must instead leverage dynamic, internal model representations tailored to the deployment context.

If we can reliably flag risky latent states at the inference boundary—especially for highly predictable input-handling vulnerabilities—we can block insecure code generations before they ever touch a repository. We need to stop treating LLMs as deterministic compilers and start building runtime security architectures that treat prompt fragility as an active runtime threat.