Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

Source

arXiv

arxiv.org/abs/2607.15263v1

Read the source

Source

Plain-Language Brief

The authors propose a new approach for evaluating security agents, focusing on cost-effectiveness rather than just success rate. They evaluate language models on offensive and defensive tasks, comparing performance at fixed cost levels instead of only reporting best-case success. The results show that offensive CTF performance improves with additional test-time compute, while defensive SOC investigation does not scale in the same way. The authors argue that security benchmarks should measure economic efficiency and operational fit along with task success.

What the Document Says

  • Security agent evaluations typically focus on peak offensive capability under generous inference budgets.
  • The study evaluates language-model security agents through a cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges.
  • Instead of reporting only best-case success, the study compares models at fixed cost levels and decomposes performance by inference spend and tool spend.
  • The results show distinct scaling regimes for red- and blue-team tasks.
  • Offensive CTF performance improves with additional test-time compute.
  • Scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive in offensive tasks.
  • Defensive SOC investigation does not scale in the same way as offensive tasks, with success depending more on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone.
  • The authors argue that security benchmarks should measure economic efficiency and operational fit alongside task success.
  • The study presents an interactive website for their results at https://evals.frontier.security.

Key Concepts Mentioned

  • Security agents
  • Offensive tasks
  • Defensive tasks
  • Cost-aware evaluation
  • Cybench challenges
  • Splunk BOTS v1
  • Inference spend
  • Tool spend

Claims Made by the Document

  • Offensive CTF performance improves with additional test-time compute.
  • Scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive in offensive tasks.

Evidence Used

  • Results from evaluating language-model security agents on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Our results show distinct scalingregimes for red- and blue-team tasks. Offensive CTF performance improves with additional test-time compute, and scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigation does not scale in the same way: success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone. We argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve. We present an interactive website with our results https://evals.frontier.security.