CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Xingyu Yan, Tingting Dai, Antonio De Domenico, Mohamed Sana, Nicola Piovesan, Changchang Li, Bowen Liu, Kun Jiang, Mengjie Zhang, Dingcheng Shan, Jing-Cheng Pang, Chenwei Wu, Sijie Wu, Lianying Chao, Haoran Cai, Jiantao Ye, Xubin Li, Simon Mark Lucas, Xin Chen
• Venue: arXiv: cs.AI
• Published: 2026-08-12
• Collected: 2026-08-13
• URL: http://arxiv.org/abs/2608.12002v1
• PDF: https://arxiv.org/pdf/2608.12002v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

The paper introduces CTBench, a public benchmark for evaluating AI agents' ability to perform troubleshooting tasks in telecom network operations. The focus is on root cause analysis and path restoration. The authors find that while state-of-the-art agents excel at identifying endpoints in path-restoration tasks, they struggle with various operational faults, particularly interface state, link-layer, service-management issues, and more. They also note that larger resource usage does not always lead to better diagnosis.

What the Document Says

• The paper discusses the use of AI agents in automating network operations and maintenance.
• Existing evaluations are criticized for failing to accurately model real network characteristics or assess agents under diverse telecom environments.
• CTBench is introduced as a public benchmark for evaluating AI agents' troubleshooting abilities.
• CTBench focuses on root cause analysis and path restoration.
• Each task in CTBench is constructed and annotated by experts with rich metadata, including golden evidence steps.
• CTBench uses expert-grounded metrics that evaluate both final answers and diagnostic evidence.
• Experiments show that state-of-the-art agents perform well at identifying endpoints in path-restoration tasks but struggle with various operational faults.
• Agents have difficulties with interface state, link-layer, service-management, and other issues.
• Even when agents produce correct final answers, they often fail to provide evidence-grounded diagnoses.
• The results indicate that path restoration is generally more resource-expensive but larger resource usage does not guarantee better diagnosis.

Key Concepts Mentioned

• AI agents
• Telecom network operations
• Network faults
• Root cause analysis
• Path restoration
• CTBench
• Expert-grounded metrics
• Diagnostic evidence
• Operational practice
• Resource usage

Claims Made by the Document

• Existing evaluations fail to accurately model real network characteristics or assess agents under diverse telecom environments.
• CTBench can evaluate whether an agent behaves like a competent telecom troubleshooting engineer.
• State-of-the-art agents underperform in root cause analysis.
• Agents struggle with interface state, link-layer, service-management, and other operational faults.
• Even when agents produce plausible or correct final answers, they often fail to provide the evidence-grounded diagnoses required in operational practice.
• Path restoration is generally more resource-expensive but larger resource usage does not necessarily translate into better diagnosis.

Evidence Used

• The paper discusses the challenges of existing evaluations and introduces CTBench as a solution.
• CTBench is described in detail, including its focus on root cause analysis and path restoration, and its use of expert-grounded metrics.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

• The paper does not provide specific examples of the operational faults that agents struggle with.

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The text lacks information about which specific state-of-the-art agents were tested and their performance beyond path restoration.

Open Questions

• What specific operational faults were encountered during the experiments?
• Were there any state-of-the-art agents that performed well in root cause analysis?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The text provides details about the struggles of agents in various operational faults, but it doesn't list specific examples.

Raw Abstract / Excerpt

Agents are increasingly considered for automating network operations and maintenance, where engineers must diagnose network faults, optimize configurations to enhance services, and reduce operational costs while acting under strict constraints. However, existing evaluations fail to accurately model real network characteristics or assess agents under partially observable telecom environments with diverse vendors, devices, protocols, and interfaces. In this paper, we introduce CTBench, a public benchmark for assessing whether an agent behaves like a competent telecom troubleshooting engineer. CTBench focuses on root cause analysis and path restoration. Each task is constructed by experts and annotated with rich task metadata, including golden evidence steps. CTBench uses expert-grounded metrics that evaluate both final answers and the diagnostic evidence. Experiments with representative harness-model combinations show that state-of-the-art agents perform very well at identifying endpoints in path-restoration tasks but, more generally, underperform in root cause analysis. In particular, agents struggle with interface state, link-layer, service-management, and other operational faults. Most importantly, even when agents produce plausible or correct final answers, they often fail to provide the evidence-grounded diagnoses required in operational practice. Our results further show that path restoration is generally more resource expensive, yet larger resource usage does not necessarily translate into better diagnosis.