OpenRCA 2.0: From Outcome Labels to Causal Process Supervision

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Aoyang Fang, Yifan Yang, Jin'ao Shang, Qisheng Lu, Junjielung Xu, Rui Wang, Songhan Zhang, Yuzhong Zhang, Boxi Yu, Pinjia He
• Venue: arXiv: cs.AI
• Published: 2026-06-25
• Collected: 2026-06-26
• URL: http://arxiv.org/abs/2606.27154v1
• PDF: https://arxiv.org/pdf/2606.27154v1
• DOI: UNKNOWN

Plain-Language Brief

The paper introduces OpenRCA 2.0, a new benchmark for root cause analysis (RCA) of Large Language Models (LLMs). The new dataset, OpenRCA 2.0, includes 500 instances and is the first cross-system RCA benchmark with step-wise causal annotations for LLMs. The researchers found that existing LLMs fail to recover the exact root cause in most cases, and they identify a failure mode called 'ungrounded diagnosis' where LLMs can correctly identify a root cause but cannot ground it in a verified causal propagation path to the observed symptom.

What the Document Says

• OpenRCA 2.0 is introduced as a new benchmark for RCA of LLMs with step-wise causal annotations (500 instances).
• The dataset, OpenRCA 2.0, is the first cross-system RCA benchmark.
• Across 11 frontier LLMs, recovering the exact root-cause set succeeds in only 20.7% of cases on average.

Key Concepts Mentioned

• root cause analysis
• LLM agents
• OpenRCA 2.0
• PAVE
• step-wise labeling protocol
• causal propagation paths
• fault injection

Claims Made by the Document

• OpenRCA 2.0 is the first cross-system RCA benchmark with step-wise causal annotations for LLM agents.

Evidence Used

• The mechanism is forward verification: reasoning from cause to effect rather than inferring backward from symptoms.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Root cause analysis (RCA) poses a holistic test of LLM agentic capabilities, such as long-context understanding, multi-step reasoning, and tool use. However, existing datasets suffer from a fundamental gap: they label only the root cause, not the propagation path connecting it to the observed symptom, which largely simplifies the task to naive pattern matching. To support rigorous evaluation, we introduce PAVE, a step-wise labeling protocol that leverages known interventions from fault injection to reconstruct causal propagation paths. The mechanism is forward verification: reasoning from cause to effect rather than inferring backward from symptoms. Applying PAVE yields OpenRCA 2.0 (500 instances), the first cross-system RCA benchmark with step-wise causal annotations for LLM agents. Across 11 frontier LLMs, recovering the exact root-cause set succeeds in only 20.7% of cases on average. To locate where this difficulty lies, we relax the criterion and find what we call the ungrounded diagnosis: agents identify at least one correct root-cause service in 76.0% of cases, but ground that service in a verified causal propagation path to the observed symptom in only 61.5%. Outcome-only evaluation hides this failure mode; step-wise causal ground truth is the missing piece for trustworthy LLM-based RCA agents.