Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

Image generated by AI

TLDR

• What: A cost-success evaluation framework for AI security agents revealing that while offensive capabilities scale steadily with test-time compute, defensive threat-hunting efficacy requires tight tool discipline and is highly prone to benchmark contamination.
• Who's at risk: Enterprise Security Operations Centers (SOCs) deploying autonomous LLM-based incident response agents (e.g., integrated with Splunk) without cost caps or contamination controls.
• Key number: DeepSeek v4 Flash achieved a 73.0% score on BOTS v1 while racking up an astonishing 4,450 tool calls, whereas Claude Opus 4.8 scored 93.9% with only 603 tool calls—demonstrating that defensive agent success does not scale with tool invocation volume.

LLM-powered agents are transitioning from experimental sandboxes into critical security roles, including autonomous incident response pipelines and threat hunting. However, evaluating these agents solely on their peak success rates under unlimited budgets creates an operational blind spot. Every reasoning token, API query, and telemetry search costs real money, meaning that the true utility of an agentic security deployment is defined by its economic efficiency and disciplined tool usage under constrained operating budgets.

Threat Model

Background / Problem Setup

Traditional security evaluations evaluate peak capabilities under generous, often unrealistic, inference budgets. This leads to models that recursively brute-force vulnerabilities or spam security tools without regard for cost. In operational environments, every query to external APIs (e.g., VirusTotal, WHOIS) or SIEM indexing layers costs computational or financial capital.

The authors evaluate this trade-off using a common evaluation harness on two primary benchmarks: Cybench (offensive CTFs) and Splunk Boss of the SOC (BOTS) v1 (defensive log investigation).

Methodology

The evaluation utilizes the Inspect framework (UK AI Security Institute, 2024) with a ReAct-style agent. To handle long, multi-step runs without blowing past model limits, the authors implement an auto-compaction loop that triggers when context reaches 90% of the model's context window.

Offensive Evaluation: Cybench

• Dataset: Cybench hard-variant (39 professional CTF challenges from platforms like HackTheBox).
• Tools: bash, python, and a submit tool (all modeled as zero-cost).
• Metric: Mean correctness over 3 independent runs.

Defensive Evaluation: BOTS v1

• Dataset: 31 scored questions from Splunk BOTS v1, representing real threat-hunting scenarios (Po1s0n1vy web defacement and Cerber ransomware at Wayne Enterprises).
• Tools: Splunk query tools (SPL), Splunk event drill-down, Brave Search ($0.005/request), WhoisXMLAPI preview ($0.0258/request), WhoisXMLAPI full purchase ($1.29/request), VirusTotal (free), and DNS (free).
• Metric: bots_points (accounts for penalty-deductions when hint sequences are triggered by incorrect answers).

The agent's cost-success dynamics are audited retrospectively using a cost tracker that simulates hard resource cuts at specific spending thresholds:

# Conceptual implementation of retrospective cost cap evaluation
def evaluate_under_cap(trace_events, cost_cap):
    cumulative_cost = 0.0
    for event in trace_events:
        cumulative_cost += event.token_cost + event.tool_fee
        if cumulative_cost > cost_cap:
            return 0.0, "Aborted: Budget Exceeded"
    return trace_events.final_score, "Completed"

Key Results

1. Offensive Cybench Performance

As shown in Table 2, offensive success scales steadily with budget. GPT-5.5 stands out as the most cost-efficient offensive model, solving 94.1% of the challenges at $1.16 per solved equivalent. DeepSeek v4 Flash demonstrates high adaptability: limiting its budget retrospectively to $0.80 yields a 76.1% success rate, which climbs to 86.4% when given a full $2.10 budget.

Conversely, Claude Fable 5 failed to complete any challenges (0.0% success rate) due to overactive policy filtering—refusing all 117 sample-epochs before executing a single tool call.

2. Defensive BOTS v1 Performance

On defensive tasks, extra budget does not buy better performance. As detailed in Table 3, Claude Opus 4.8 is the top defensive model, securing 93.9% of BOTS points at $2.98 per 1,000 points, requiring only 603 total tool calls. In contrast, DeepSeek v4 Flash at the $4.20 cap spams 4,938 tool calls only to plateau at a lower BOTS score of 73.9%. High tool volume correlates with poor reasoning discipline, not high capability.

3. Contamination Controls: The Danger of Public SOC Benchmarks

To determine if these scores reflect true security investigation skills, the authors performed "no-tools" control probes (Table 4).

The findings are notable: Claude Opus 4.8 scored 50.0% on BOTS v1 without ever querying Splunk, web search, or any logs. Under the same conditions, GPT-5.5 scored 54.9%. This reveals massive benchmark contamination—the models are recalling answers directly from pre-training data containing public BOTS writeups.

Limitations & Open Questions

• Public Dataset Leakage: Because BOTS v1 (2017) is widely public, absolute model scores are highly inflated. The benchmark is better suited as a reproducible testbed for cost-accounting mechanics than as a clean measurement of real-world SOC threat-hunting ability.
• Message Limit Truncation: High-volume runs used a hard 250-message cap per run, which was triggered in several DeepSeek v4 runs. This truncation may artificially suppress high-volume model scores.
• Narrow Incident Scope: The defensive analysis only evaluates Splunk BOTS v1. Scaling these cost-success metrics to newer and cloud-native workloads (like AWS-heavy BOTS v3) is still an open challenge.

What Practitioners Should Do

1. Build Multi-Dimensional Budget Caps into Agent Frameworks

Do not let your agents run with unrestricted API keys. Force your runtime wrapper to terminate if total execution costs cross a strict threshold:

# Example Inspect configuration block for cost-control
agent:
  name: react_security_agent
  max_cost_limit_usd: 1.50
  max_tool_calls: 50
  compaction_trigger_ratio: 0.90

2. Implement "No-Tools" Baselines in Your Evaluations

Before trusting an LLM agent evaluation score on your internal datasets, run a control test. Ask the model the security questions without providing access to your SIEM, logs, or external tools. If the model achieves a high score, your benchmark is leaked, and you are evaluating training memory rather than investigation capability.

3. Implement Strict Query Filtering and Rate Penalties

To prevent agents from recursively calling costly external endpoints, build routing gates. Programmatically limit expensive tool calls to 3-5 invocations per run, forcing the agent to rely on cheaper local context summaries first.

4. Rely on Structured Context Compression

Instead of feeding raw, verbose log feeds directly to the context window, implement provider-native prompt caching and auto-compaction steps when context usage exceeds 90%. This reduces model token consumption cost and prevents latency spikes during long multi-step runs.

The Takeaway

Security-agent performance is not a simple game of chasing high success rates. While offensive agents benefit from extra test-time compute, defensive threat-hunting demands strict tool discipline, precise search coordination, and budget awareness. Building a cost-aware security center requires moving away from raw parameter scaling and toward highly disciplined agent architectures designed to maximize security value per dollar spent.

Den's Take

I love that this paper targets the ridiculous "success-at-all-costs" metric that plagues agentic benchmarks. In a real-world Security Operations Center (SOC), nobody is going to tolerate an autonomous agent that spams SIEM queries without constraint.

The paper’s contrast between models on the BOTS v1 defensive benchmark is highly revealing. Seeing DeepSeek v4 Flash rack up an astonishing 4,450 tool calls to hit a mediocre 73.0% success rate, while Claude Opus 4.8 achieved 93.9% with just 603 calls, exposes a massive gap in tool discipline. When you are operating under tight budgets ranging from $0.75 to $4.20 per sample run, efficiency is not just a nice-to-have; it is the entire game. While an offensive agent in Cybench might scale steadily with test-time compute, a defensive agent must do the opposite—it requires precise, targeted reasoning to avoid API fatigue.

This practical tension closely mirrors issues with context bloating, where noisy log environments quickly derail LLM-based threat hunting. If you are deploying agents on frameworks like Inspect, you cannot just measure whether they eventually find the flag; you must track what it actually cost you to get there.