CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations

Source

Source name: arXiv
Source kind: paper
Authors / organization: Xingyu Yan, Tingting Dai, Antonio De Domenico, Mohamed Sana, Nicola Piovesan, Changchang Li, Bowen Liu, Kun Jiang, Mengjie Zhang, Dingcheng Shan, Jing-Cheng Pang, Chenwei Wu, Sijie Wu, Lianying Chao, Haoran Cai, Jiantao Ye, Xubin Li, Simon Mark Lucas, Xin Chen
Venue: arXiv: cs.AI
Published: 2026-08-12
Collected: 2026-08-13
URL: http://arxiv.org/abs/2608.12002v1
PDF: https://arxiv.org/pdf/2608.12002v1
DOI: UNKNOWN
Registry group: academic_apis_and_preprints
Evidence role: canonical_academic_index
Language: en
Country / region: UNKNOWN

Plain-Language Brief

Researchers have developed a new benchmark called CTBench to evaluate the performance of AI agents in troubleshooting network issues in telecom environments. The benchmark focuses on root cause analysis and path restoration, and it includes tasks constructed by experts with rich metadata and golden evidence steps. Experiments showed that state-of-the-art agents perform well at identifying endpoints for path-restoration tasks but struggle with various types of faults, particularly interface state, link-layer, service-management, and operational faults. The study also found that while agents may produce correct answers, they often fail to provide evidence-based diagnoses required in practical operations.

What the Document Says

CTBench is a new benchmark for evaluating AI agents' performance in network troubleshooting in telecom environments.
The focus of CTBench is on root cause analysis and path restoration.
Tasks in CTBench are constructed by experts and annotated with rich metadata, including golden evidence steps.
CTBench uses expert-grounded metrics to evaluate both final answers and diagnostic evidence.
Experiments with representative harness-model combinations showed that state-of-the-art agents perform well at identifying endpoints for path-restoration tasks but struggle with various types of faults, particularly interface state, link-layer, service-management, and operational faults.
The study also found that while agents may produce correct answers, they often fail to provide evidence-based diagnoses required in practical operations.

Key Concepts Mentioned

AI agents
network troubleshooting
telecom environments
root cause analysis
path restoration
CTBench
expert-grounded metrics
state-of-the-art agents
interface state
link-layer
service-management
operational faults

Claims Made by the Document

CTBench can accurately assess AI agents' performance in troubleshooting network issues.
State-of-the-art agents perform well at identifying endpoints for path-restoration tasks.
State-of-the-art agents struggle with various types of faults, particularly interface state, link-layer, service-management, and operational faults.

Evidence Used

Experiments with representative harness-model combinations

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Agents are increasingly considered for automating network operations and maintenance, where engineers must diagnose network faults, optimize configurations to enhance services, and reduce operational costs while acting under strict constraints. However, existing evaluations fail to accurately model real network characteristics or assess agents under partially observable telecom environments with diverse vendors, devices, protocols, and interfaces. In this paper, we introduce CTBench, a public benchmark for assessing whether an agent behaves like a competent telecom troubleshooting engineer. CTBench focuses on root cause analysis and path restoration. Each task is constructed by experts and annotated with rich task metadata, including golden evidence steps. CTBench uses expert-grounded metrics that evaluate both final answers and the diagnostic evidence. Experiments with representative harness-model combinations show that state-of-the-art agents perform very well at identifying endpoints in path-restoration tasks but, more generally, underperform in root cause analysis. In particular, agents struggle with interface state, link-layer, service-management, and other operational faults. Most importantly, even when agents produce plausible or correct final answers, they often fail to provide the evidence-grounded diagnoses required in operational practice. Our results further show that path restoration is generally more resource expensive, yet larger resource usage does not necessarily translate into better diagnosis.