Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

OpenRCA 2.0: From Outcome Labels to Causal Process Supervision

Source

arXiv

arxiv.org/abs/2606.27154v1

Read the source

Source

Plain-Language Brief

The paper introduces OpenRCA 2.0, a new benchmark for root cause analysis (RCA) that includes step-wise causal annotations for large language models (LLMs). The benchmark aims to test the capabilities of LLMs in long-context understanding, multi-step reasoning, and tool use. The authors find that existing datasets have a fundamental gap: they only label the root cause, not the propagation path connecting it to the observed symptom. OpenRCA 2.0 addresses this by using a step-wise labeling protocol called PAVE, which reconstructs causal propagation paths through forward verification (reasoning from cause to effect). The authors find that LLMs struggle to recover the exact root cause set and ground that service in a verified causal propagation path to the observed symptom. They argue that outcome-only evaluation hides this failure mode and that step-wise causal ground truth is essential for trustworthy LLM-based RCA agents.

What the Document Says

  • Root cause analysis (RCA) is a test of large language model (LLM) agentic capabilities.
  • Existing datasets for RCA only label the root cause, not the propagation path connecting it to the observed symptom.
  • PAVE is a step-wise labeling protocol that reconstructs causal propagation paths through forward verification.
  • OpenRCA 2.0 is a new benchmark for RCA with step-wise causal annotations for LLMs, created using PAVE.
  • OpenRCA 2.0 has 500 instances.
  • The benchmark is the first cross-system RCA benchmark.
  • Across 11 frontier LLMs, recovering the exact root-cause set succeeds in only 20.7% of cases on average.
  • Agents identify at least one correct root-cause service in 76.0% of cases, but ground that service in a verified causal propagation path to the observed symptom in only 61.5%.
  • Outcome-only evaluation hides the failure mode of LLMs in RCA.
  • Step-wise causal ground truth is essential for trustworthy LLM-based RCA agents.

Key Concepts Mentioned

  • root cause analysis
  • large language models
  • LLM agentic capabilities
  • PAVE
  • OpenRCA 2.0
  • step-wise labeling protocol
  • causal propagation paths
  • forward verification
  • frontier LLMs

Claims Made by the Document

  • Existing RCA datasets only label the root cause, not the propagation path.
  • PAVE can reconstruct causal propagation paths for RCA.
  • OpenRCA 2.0 is a new benchmark with step-wise causal annotations for LLMs.
  • LLMs struggle to recover the exact root cause set.
  • LLMs identify at least one correct root-cause service in most cases, but grounding that service in a verified causal path is challenging.

Evidence Used

  • The paper describes the limitations of existing RCA datasets and introduces OpenRCA 2.0 as a solution.
  • The paper explains how PAVE works and its application to OpenRCA 2.0.
  • The paper provides statistics about the success rate of LLMs in recovering root cause sets on OpenRCA 2.0.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Root cause analysis (RCA) poses a holistic test of LLM agentic capabilities, such as long-context understanding, multi-step reasoning, and tool use. However, existing datasets suffer from a fundamental gap: they label only the root cause, not the propagation path connecting it to the observed symptom, which largely simplifies the task to naive pattern matching. To support rigorous evaluation, we introduce PAVE, a step-wise labeling protocol that leverages known interventions from fault injection to reconstruct causal propagation paths. The mechanism is forward verification: reasoning from cause to effect rather than inferring backward from symptoms. Applying PAVE yields OpenRCA 2.0 (500 instances), the first cross-system RCA benchmark with step-wise causal annotations for LLM agents. Across 11 frontier LLMs, recovering the exact root-cause set succeeds in only 20.7% of cases on average. To locate where this difficulty lies, we relax the criterion and find what we call the ungrounded diagnosis: agents identify at least one correct root-cause service in 76.0% of cases, but ground that service in a verified causal propagation path to the observed symptom in only 61.5%. Outcome-only evaluation hides this failure mode; step-wise causal ground truth is the missing piece for trustworthy LLM-based RCA agents.