Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Paul Kassianik, Blaine Nelson, Yaron Singer
• Venue: arXiv: cs.CR
• Published: 2026-07-16
• Collected: 2026-07-17
• URL: http://arxiv.org/abs/2607.15263v1
• PDF: https://arxiv.org/pdf/2607.15263v1
• DOI: UNKNOWN

Plain-Language Brief

The authors propose a new approach for evaluating security agents by focusing on their cost-effectiveness instead of just their success rate in offensive and defensive tasks. They demonstrate this using language models on Cybench challenges and Splunk BOTS v1 investigation challenges, comparing models based on their inference spend and tool spend at fixed cost levels. The results show different scaling regimes for red-team and blue-team tasks, with offensive CTF performance improving with more compute, while defensive SOC investigation does not scale the same way and relies more on disciplined tool use and enrichment. They argue that security agent benchmarks should also measure economic efficiency and operational fit.

What the Document Says

• The paper discusses a new evaluation method for language-model security agents that focuses on cost-effectiveness instead of just success rate in offensive and defensive tasks.
• They use Cybench challenges and Splunk BOTS v1 investigation challenges to demonstrate this approach.
• They compare models based on their inference spend and tool spend at fixed cost levels.
• Results show distinct scaling regimes for red-team (offensive) and blue-team (defensive) tasks.
• Offensive CTF performance improves with additional test-time compute, while defensive SOC investigation does not scale the same way.
• They argue that security agent benchmarks should measure economic efficiency and operational fit alongside task success.
• An interactive website with their results is presented at https://evals.frontier.security.

Key Concepts Mentioned

• language-model security agents
• cost-aware evaluation
• offensive Cybench challenges
• defensive Splunk BOTS v1 investigation challenges
• inference spend
• tool spend
• red-team tasks
• blue-team tasks
• economic efficiency
• operational fit

Claims Made by the Document

• A cost-aware approach is a useful evaluation method for language-model security agents.
• Offensive CTF performance improves with additional test-time compute.
• Defensive SOC investigation does not scale the same way as offensive tasks.
• Security agent benchmarks should measure economic efficiency and operational fit.

Evidence Used

• Results from Cybench challenges and Splunk BOTS v1 investigation challenges
• Comparisons between models based on inference spend and tool spend at fixed cost levels

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Our results show distinct scalingregimes for red- and blue-team tasks. Offensive CTF performance improves with additional test-time compute, and scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigation does not scale in the same way: success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone. We argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve. We present an interactive website with our results https://evals.frontier.security.