Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Paul Kassianik, Blaine Nelson, Yaron Singer
• Venue: arXiv: cs.CR
• Published: 2026-07-16
• Collected: 2026-07-17
• URL: http://arxiv.org/abs/2607.15263v1
• PDF: https://arxiv.org/pdf/2607.15263v1
• DOI: UNKNOWN

Plain-Language Brief

The research focuses on evaluating the efficiency and practicality of language model security agents in both offensive and defensive tasks, rather than just their peak offensive capability. The study uses cost-aware evaluation methods on Cybench challenges for offense and Splunk BOTS v1 investigation challenges for defense. Results show that offensive CTF performance improves with additional compute resources, while defensive SOC investigation does not scale in the same way. The researchers argue that future security agent benchmarks should measure economic efficiency and operational fit alongside task success.

What the Document Says

• The research evaluates language model security agents using cost-aware methods on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges.
• Offensive CTF performance improves with additional compute resources.
• Defensive SOC investigation does not scale in the same way as offensive tasks.
• The study compares models at fixed cost levels and decomposes performance by inference spend and tool spend.

Key Concepts Mentioned

• security agents
• offensive Cybench challenges
• defensive Splunk BOTS v1 investigation challenges
• cost-aware evaluation
• inference spend
• tool spend
• SOC investigation
• economic efficiency

Claims Made by the Document

• Offensive CTF performance improves with additional compute resources.
• Defensive SOC investigation does not scale in the same way as offensive tasks.

Evidence Used

• The research findings based on the experiments conducted on Cybench challenges for offense and Splunk BOTS v1 investigation challenges for defense.

Methods / Approach

• Cost-aware evaluation methods
• Comparison at fixed cost levels
• Decomposition of performance by inference spend and tool spend.

Limitations Stated by the Authors

• The study may not cover all aspects of security agents' capabilities or limitations.
• The research is based on specific challenges (Cybench and Splunk BOTS v1) and may not reflect performance in other scenarios.

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The agent may not fully understand the nuances of security-agent benchmarks and their implications.
• The agent may not be able to interpret the authors' arguments about operational fit and economic efficiency in full detail.

Open Questions

• What are the specific tools and enrichment requests used in the study?
• How does the performance of security agents vary across different types of security tasks?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The text does not provide details about the specific tools and enrichment requests used in the study.
• The research focuses on offensive CTF and defensive SOC investigation challenges, but it's unclear how the performance of security agents varies across different types of security tasks.

Raw Abstract / Excerpt

Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Our results show distinct scalingregimes for red- and blue-team tasks. Offensive CTF performance improves with additional test-time compute, and scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigation does not scale in the same way: success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone. We argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve. We present an interactive website with our results https://evals.frontier.security.