Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

OpenRCA 2.0: From Outcome Labels to Causal Process Supervision

Source

arXiv

arxiv.org/abs/2606.27154v1

Read the source

Source

Plain-Language Brief

The paper introduces OpenRCA 2.0, a new benchmark for root cause analysis (RCA) of large language models (LLMs). The benchmark includes step-wise causal annotations for LLMs and addresses the issue that existing datasets only label the root cause, not the path connecting it to the observed symptom. The authors find that, on average, LLMs can successfully identify the exact root cause set in only 20.7% of cases, with a higher rate (76.0%) of identifying at least one correct root-cause service, but failing to ground that service in a verified causal path to the observed symptom in 38.5% of cases.

What the Document Says

  • The paper presents OpenRCA 2.0, a new benchmark for RCA of LLMs.
  • OpenRCA 2.0 contains step-wise causal annotations for LLMs.
  • Existing datasets only label the root cause, not the path connecting it to the observed symptom.
  • PAVE, a step-wise labeling protocol is introduced, which uses known interventions from fault injection to reconstruct causal propagation paths.
  • Applying PAVE yields OpenRCA 2.0 with 500 instances.
  • OpenRCA 2.0 is the first cross-system RCA benchmark with step-wise causal annotations for LLMs.
  • Across 11 frontier LLMs, recovering the exact root-cause set succeeds in only 20.7% of cases on average.
  • LLMs identify at least one correct root-cause service in 76.0% of cases.
  • However, grounding that service in a verified causal propagation path to the observed symptom is successful in only 61.5% of cases.
  • Outcome-only evaluation hides failure modes; step-wise causal ground truth is needed for trustworthy LLM-based RCA agents.

Key Concepts Mentioned

  • Root cause analysis
  • Large language models (LLMs)
  • OpenRCA 2.0
  • PAVE
  • causal propagation paths
  • fault injection

Claims Made by the Document

  • Existing datasets only label the root cause, not the path connecting it to the observed symptom.
  • OpenRCA 2.0 is the first cross-system RCA benchmark with step-wise causal annotations for LLMs.
  • Across 11 frontier LLMs, recovering the exact root-cause set succeeds in only 20.7% of cases on average.

Evidence Used

  • The paper presents a new benchmark with step-wise causal annotations.
  • The text mentions that existing datasets lack complete causal information.
  • The results from the experiments are presented, showing the success rates for various LLMs.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Root cause analysis (RCA) poses a holistic test of LLM agentic capabilities, such as long-context understanding, multi-step reasoning, and tool use. However, existing datasets suffer from a fundamental gap: they label only the root cause, not the propagation path connecting it to the observed symptom, which largely simplifies the task to naive pattern matching. To support rigorous evaluation, we introduce PAVE, a step-wise labeling protocol that leverages known interventions from fault injection to reconstruct causal propagation paths. The mechanism is forward verification: reasoning from cause to effect rather than inferring backward from symptoms. Applying PAVE yields OpenRCA 2.0 (500 instances), the first cross-system RCA benchmark with step-wise causal annotations for LLM agents. Across 11 frontier LLMs, recovering the exact root-cause set succeeds in only 20.7% of cases on average. To locate where this difficulty lies, we relax the criterion and find what we call the ungrounded diagnosis: agents identify at least one correct root-cause service in 76.0% of cases, but ground that service in a verified causal propagation path to the observed symptom in only 61.5%. Outcome-only evaluation hides this failure mode; step-wise causal ground truth is the missing piece for trustworthy LLM-based RCA agents.