Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Paul Kassianik, Blaine Nelson, Yaron Singer
• Venue: arXiv: cs.CR
• Published: 2026-07-16
• Collected: 2026-07-17
• URL: http://arxiv.org/abs/2607.15263v1
• PDF: https://arxiv.org/pdf/2607.15263v1
• DOI: UNKNOWN

Plain-Language Brief

The researchers evaluate the performance of language model security agents, focusing on costs rather than just success rates. They use offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges to compare models based on inference spend and tool spend at fixed cost levels. The study shows that offensive CTF performance improves with additional compute resources, while defensive SOC investigations depend more on disciplined tool usage and selective enrichment. They argue for the need to measure economic efficiency and operational fit along with task success in security-agent benchmarks.

What the Document Says

• Researchers evaluate language model security agents based on cost-success.
• They use offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges.
• Performance is compared at fixed cost levels, considering inference spend and tool spend.
• Offensive CTF performance improves with additional compute resources.
• Defensive SOC investigations depend more on disciplined tool usage and selective enrichment than raw reasoning budget alone.

Key Concepts Mentioned

• Language model security agents
• Offensive Cybench challenges
• Defensive Splunk BOTS v1 investigation challenges
• Inference spend
• Tool spend
• Cost-success
• SOC investigations

Claims Made by the Document

• Offensive CTF performance improves with additional compute resources.
• Defensive SOC investigations do not scale in the same way.

Evidence Used

• Results from experiments on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

The authors acknowledge that their research may not fully capture the complexity and diversity of operational security scenarios.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The agent's responses are based on the provided text, and it may not fully understand the nuances or implications of the study.

Open Questions

• What specific metrics are used to measure economic efficiency in the study?
• How does the performance of other types of security challenges scale with compute resources?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The study suggests that offensive CTF performance improves with more compute resources, but it's unknown if the same trend holds for other types of security challenges.
• The paper argues for cost-aware evaluations in security agents. However, it doesn't mention what specific metrics they use to measure economic efficiency.

Raw Abstract / Excerpt

Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Our results show distinct scalingregimes for red- and blue-team tasks. Offensive CTF performance improves with additional test-time compute, and scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigation does not scale in the same way: success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone. We argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve. We present an interactive website with our results https://evals.frontier.security.