Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

Source

arXiv

arxiv.org/abs/2607.15263v1

Read the source

Source

Plain-Language Brief

The paper suggests that traditional methods for evaluating security agents, focusing on their offensive capabilities under generous inference budgets, are insufficient as they don't account for the cost of each reasoning step, tool call, or telemetry query. The authors propose a cost-aware evaluation method and apply it to language model security agents on Cybench challenges for offense and Splunk BOTS v1 investigation challenges for defense. They find that offensive CTF performance improves with additional compute resources, while defensive SOC investigations depend more on disciplined tool use, telemetry navigation, and selective enrichment than raw reasoning budget alone. The authors argue for cost-aware evaluations to better understand which models are practically useful today and where defensive agents still need improvement.

What the Document Says

  • Security agent evaluations typically focus on peak offensive capability under generous inference budgets.
  • The paper proposes a cost-aware evaluation method for language model security agents.
  • This method is applied to Cybench challenges for offense and Splunk BOTS v1 investigation challenges for defense.
  • Offensive CTF performance improves with additional test-time compute resources.
  • Scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive in offensive CTF.
  • Defensive SOC investigations do not scale in the same way as offensive tasks, relying more on disciplined tool use, telemetry navigation, and selective enrichment than raw reasoning budget alone.
  • The authors argue for security agent benchmarks to measure economic efficiency and operational fit alongside task success.

Key Concepts Mentioned

  • security agents
  • evaluation methods
  • cost-aware evaluation
  • Cybench challenges
  • Splunk BOTS v1 investigation challenges
  • offensive CTF
  • defensive SOC investigations

Claims Made by the Document

  • Traditional security agent evaluations are insufficient as they don't account for costs.
  • Cost-aware evaluation provides a clearer picture of practical usefulness and areas for improvement in security agents.
  • Offensive CTF performance improves with additional compute resources.
  • Defensive SOC investigations rely more on disciplined tool use, telemetry navigation, and selective enrichment than raw reasoning budget alone.

Evidence Used

  • The paper discusses the traditional methods of security agent evaluations and their limitations.
  • The authors present a new cost-aware evaluation method for language model security agents.
  • The results from applying this method to Cybench and Splunk BOTS v1 challenges are presented.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

The paper may not provide sufficient details about the specific methods and models used in the study.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The abstract does not contain enough information to fully understand the methodology, data sources, or conclusions drawn in the research.

Open Questions

  • What specific language model security agents were used in the study?
  • What is the exact methodology for the cost-aware evaluation?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The text does not specify which particular language model security agents were used.
  • Details about the exact methodology for the cost-aware evaluation are not provided in the abstract.

Raw Abstract / Excerpt

Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Our results show distinct scalingregimes for red- and blue-team tasks. Offensive CTF performance improves with additional test-time compute, and scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigation does not scale in the same way: success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone. We argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve. We present an interactive website with our results https://evals.frontier.security.