SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response

Source

Source name: arXiv
Source kind: paper
Authors / organization: Lehan Wang, Boli Chen, Ruixue Ding, Pengjun Xie, Jinwei Huang, Zhendong Liu, Shuo Wang, Tao Lei, Xin Ouyang, Xiaomeng Li
Venue: arXiv: cs.CR
Published: 2026-07-29
Collected: 2026-07-30
URL: http://arxiv.org/abs/2607.26791v1
PDF: https://arxiv.org/pdf/2607.26791v1
DOI: UNKNOWN

Plain-Language Brief

The study introduces SecRespond, a new benchmark for evaluating AI agents in post-compromise incident response scenarios. The benchmark tests the ability of agents to produce forensic reports and remediation plans given a snapshot of compromised hosts, alerts, vulnerability scans, and baseline checks. The study finds that while current agents can detect problems exposed by alerts, they struggle with proactive investigation and comprehensive remediation.

What the Document Says

The study presents SecRespond, a benchmark for evaluating AI agents in post-compromise incident response.
SecRespond requires agents to produce forensic reports on intrusions, baseline risks, and vulnerability risks, as well as remediation plans given a compromised host snapshot, alerts, scans, and checks.
The benchmark includes 10 cyber ranges built from different compromised cloud hosts, with various entry-point types, ATT&CK techniques, and operating systems.
23 frontier LLMs are evaluated on the OpenCode agent harness in this study.
Experimental results show that current agents can reliably detect problems from alerts but struggle with proactive investigation and comprehensive remediation. No model achieves complete detection and remediation on any single range.
SecRespond is available publicly at https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond.

Key Concepts Mentioned

Large Language Model (LLM) agents
post-compromise incident response
cybersecurity benchmark
SecRespond
forensic reports
remediation plan
compromised hosts
alerts
vulnerability scans
baseline checks

Claims Made by the Document

Current AI agents can reliably detect problems exposed by alerts but struggle with proactive investigation and comprehensive remediation.
No current model achieves complete detection and remediation on any single range.

Evidence Used

Experimental results show that current agents can reliably uncover the problems exposed by alerts, but they struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans. No model achieves complete detection and remediation on any single range.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities. However, existing cybersecurity benchmarks focus on pre-compromise settings where agents are placed in a clean and idealized environment before an attack occurs. This leaves the post-compromise setting underexplored. To address this gap, we introduce SecRespond, the first benchmark for evaluating LLM agents on the post-compromise incident-response workflow. Given a forensic disk snapshot of a compromised host together with the alerts, vulnerability scans, and baseline checks reported by a host security product, agents are required to produce forensic reports on intrusions, baseline risks, and vulnerability risks, together with a remediation plan. We instantiate this task across 10 cyber ranges, each constructed from a distinct compromised cloud host, spanning 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. We evaluate 23 frontier LLMs on the OpenCode agent harness. Experimental results show that although current agents can reliably uncover the problems exposed by alerts, they struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans, with no model achieving complete detection and remediation on any single range. This reveals a fundamental bottleneck in building agents for real-world incident response. The benchmark is publicly available at https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond.