Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming

Image generated by AI

TLDR

• What: AHA (Agent Hacks Agents) is an automated red-teaming framework that uses a falsifiable research loop to discover structured, reusable "vulnerability concepts" (represented as a Vulnerability Concept Graph) rather than fragile, one-off exploit payloads.
• Who's at risk: Autonomous, tool-using LLM agent environments (e.g., Claude Code, Codex, and workspace agents executing tool integrations) operating over untrusted repositories, files, or external APIs.
• Key number: Frozen vulnerability graphs discovered by AHA achieve an average Attack Success Rate (ASR) of 47.0% under a strict single-shot, zero-test-time-search protocol, beating the strongest automated red-teaming baseline by 14.2 percentage points overall.

With the rapid emergence of autonomous coding assistants and tool-using environments like Claude Code, Codex, and RAG pipelines, LLM agents are no longer just generating text—they are actively executing shell commands, modifying local file systems, and calling external APIs. As a result, a security failure in these environments is no longer a simple toxic generation; it is an active operational breach, such as unauthorized data exfiltration or silent backdoor installation. Traditional red-teaming frameworks focus heavily on optimizing fragile payload strings that break the moment a victim model is updated, patched, or swapped.

To bridge this gap, this paper introduces AHA (Agent Hacks Agents), a framework that automates the discovery of durable, reusable vulnerability knowledge. Instead of finding transient exploits, AHA behaves as an autonomous security researcher, committing to falsifiable hypotheses to map out an agent's structural weaknesses.

Threat Model

Background / Problem Setup

Existing automated red-teaming pipelines treat vulnerability discovery as a mathematical optimization task, seeking to maximize an empirical attack success signal. This often results in "judge gaming," where an exploit succeeds due to transient prompt wording or local tool behaviors rather than identifying a systematic vulnerability.

The authors compare AHA against three main paradigms in automated LLM red-teaming:

1. T-MAP (Lee et al., 2026): Uses quality-diversity evolutionary algorithms to generate a diverse archive of jailbreak prompts.
2. IterInject (Chen et al., 2026b): Employs a feedback-driven optimization loop specifically for indirect prompt injections.
3. AutoRISE (Gautam et al., 2026): Automatically codes and edits an attack program to discover exploit strategies.

Unlike these methods, which retain concrete payloads or attack programs, AHA's target artifact is a Vulnerability Concept Graph (VCG). A VCG maps out why a failure occurs by linking attacker-facing surfaces to unsafe trajectories through auditable claims, enabling conditions, and falsifiers.

Comparison of Red-Teaming Frameworks

Methodology

AHA operates via a two-stage process: Stage 1: Vulnerability Concept Discovery Loop and Stage 2: Frozen Concept Deployment and Evaluation (as shown in Figure 1).

+--------------------------------------------------------------------------+
|                        STAGE 1: DISCOVERY LOOP                           |
|                                                                          |
|  [Orchestrator] ---> [Hypothesizer] (Commits to claim & falsifier)        |
|                            |                                             |
|                            v                                             |
|                     [Attack Designer] (Instantiates scenario payload)    |
|                            |                                             |
|                            v                                             |
|                     [Sandbox Execution] (Runs payload on victim agent)   |
|                            |                                             |
|                            v                                             |
|                     [Reflector] (Classifies trace vs. falsifier)         |
|                            |                                             |
|                            v                                             |
|                     [Concept Update] (Updates the VCG)                   |
|                                                                          |
|  * A Critic audits the loop every 10 iterations to prevent reward hacks. |
+--------------------------------------------------------------------------+

Step 1: Falsifiable Hypothesis Generation

Before writing any exploit, the Hypothesizer sub-agent commits to a vulnerability claim and an explicit falsifier (what observation would prove the hypothesis false). For example, if the claim is "the agent misframes destructive tool use as routine data cleanup," the falsifier is "the model asks for human verification or rejects the deletion."

Step 2: Attack Instantiation

The Attack-Designer reads the committed hypothesis and the target task's visible metadata to generate a valid payload matching the scenario's contract (e.g., YAML edits, tool-output splices, or direct user messages).

Step 3: Sandboxed Execution & Reflection

The payload runs inside a secure, containerized Docker sandbox (CPU/memory-capped to prevent escape, with network access restricted). The Reflector sub-agent records the trajectory and judges whether the falsifier was triggered.

Step 4: Promotion to the VCG

To prevent one-off successes from entering the knowledge library, a candidate concept $c$ must clear a strict, multi-instance Evidence Rule:

This rules out any concepts that do not consistently reproduce across at least three discovery instances, or those that exhibit high falsification rates.

Key Results

AHA was evaluated on three distinct red-teaming scenarios:

• AgentHazard (direct prompt injections across 10 harm categories).
• AgentDyn (indirect prompt injections in shopping, GitHub, and daily-life environments).
• DTap (direct and indirect injections in stateful backend domains like CRMs and hospital systems).

Reusability Evaluation: Held-Out Attack Success Rate (ASR, %)

At evaluation time, the discovered VCG is frozen. For each held-out task, the concept-selection model chooses the most applicable concept and runs a single-shot attack with zero test-time search.

Note: Star (*) denotes that the baseline's discovery artifact (elite archive, seed bank, or attack program) was frozen before deployment to match AHA's single-shot testing condition.

Ablation Study: What Happens Without Loop Safeguards?

Table 4 in the paper analyzes the necessity of AHA's safeguards (the committed Falsifier, cross-episode VCG Memory, and periodic Critic) on the Minimax-M2.7 model under Claude Code:

Removing these constraints leaves the discovery-time rates artificially high, but the resulting concepts fail to generalize to unseen instances. Without a falsifier, for instance, AgentHazard's held-out ASR collapses from 75.56% to 56.67% because the model records judge-gamed or accidental successes as valid concepts.

Limitations & Open Questions

1. Tooling and Execution Flakiness: As highlighted in Footnote 2, testing tool-use agents is highly susceptible to execution-layer flakiness. Under Codex, many multi-tool DTap runs failed because the agent's tool router rejected non-deterministic, flattened tool names (due to upstream issues in openai/codex#26234).
2. High Discovery-Time Compute Costs: While executing frozen VCGs is cheap (single-shot, no test-time search), discovering these concepts requires up to 100 iterations of SOTA model calls (e.g., Claude-4.8-Opus). This makes initial mapping computationally and financially heavy.
3. Black-box Boundary Assumptions: AHA operates entirely at the black-box interface level. It does not account for internal white-box protections, such as real-time system prompt alignments, logit-level filtering, or vector-database sanitation mechanisms, which could mitigate these threats before they reach tool execution.

What Practitioners Should Do

1. Implement Strict Sanity Checks on High-Impact Tools

If your LLM agent has access to write operations, shell execution (bash), or sensitive databases (CRM, patient files), you must enforce strict, non-bypassable guardrails. Never allow the agent to issue shell commands or file updates directly without intermediate, rule-based parsing.

2. Restrict LLM Tool Authorization via Session Tokenization

Rather than giving LLM agents persistent access keys to databases, use short-lived, task-scoped credentials. If an agent tries to delete records in bulk (such as the "provenance-laundered deletion" shown in the DTap case study, where the agent deleted $1,000,000 in pipeline value), the backend API should flag and reject the request if the specific session token lacks explicit permission for destructive actions.

3. Integrate Falsifiable Hypotheses into Automated Security CI/CD

Incorporate the concept of a "Falsifier" into your internal red-teaming and evaluation suites. When testing agents, do not rely solely on whether a test pass/fail LLM evaluator reports a breach. Assert specific, deterministic trajectory side-effects (e.g., checking if a file was modified or if a network packet was sent) to verify whether the agent actually performed a forbidden step.

4. Sandbox All Agent Environments

Run your autonomous assistants in tightly restricted, ephemeral Docker containers with limited CPU, memory, and wall-clock times. Disable raw outbound network access except to verified, whitelisted API endpoints, which prevents data exfiltration routes via automated curl calls.

The Takeaway

The era of evaluating LLM safety solely through jailbreak prompt strings is coming to an end. As LLMs transition into autonomous agents running tools in production environments, safety failures become physical trajectory-level breaches. Frameworks like AHA demonstrate that to secure these agents, defenders must shift their focus from cataloging fragile strings to structurally mapping out the underlying vulnerability mechanisms that survive model updates and system patches.

Den's Take

I’ve grown incredibly tired of the endless stream of AI red-teaming papers optimizing fragile, token-level jailbreaks that break the moment a vendor pushes a minor patch. This is why AHA (Agent Hacks Agents) is a breath of fresh air. By automating the discovery of structured, reusable Vulnerability Concept Graphs (VCGs) rather than brittle exploit payloads, the authors shift the security paradigm toward mapping systematic weaknesses in autonomous, tool-using environments like Claude Code and Codex.

The empirical results are solid: achieving a 47.0% average Attack Success Rate (ASR) under a strict single-shot, zero-test-time-search protocol—beating the strongest baseline by 14.2 percentage points—proves that structured concept mapping works, even when targeting complex backends like Deepseek-V4-Pro, Kimi-K2.6, and Minimax-M2.7. This shift from fragile payload optimization to mapping structural, trajectory-level behavioral flaws highlights why runtime mitigation is so difficult, a challenge I previously explored when analyzing containment wrappers for web agents.

My practical concern is the computational footprint. Bounding the discovery loop to 28 to 100 iterations using high-end host models is reasonable for an offline audit, but it remains too slow and expensive for real-time security regression testing in CI/CD pipelines. We must find a way to distill these heavy automated research loops if we want to secure rapidly iterating agentic deployments.