Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

OpenRCA 2.0: From Outcome Labels to Causal Process Supervision

Source

arXiv

arxiv.org/abs/2606.27154v1

Read the source

Source

Plain-Language Brief

Researchers introduce a new method, called PAVE, to support the rigorous evaluation of large language models (LLMs) in root cause analysis (RCA). The current datasets only label the root cause and not the path connecting it to the observed symptom. This simplifies the task to pattern matching. PAVE uses known interventions from fault injection to reconstruct causal propagation paths, allowing for more accurate evaluation of LLMs. The researchers present OpenRCA 2.0, a new RCA benchmark with step-wise causal annotations for LLMs, which shows that existing LLMs struggle to accurately identify the root cause in many cases.

What the Document Says

  • Researchers introduce PAVE, a step-wise labeling protocol using known interventions from fault injection to reconstruct causal propagation paths for evaluating LLMs in RCA.
  • OpenRCA 2.0 is introduced, which is the first cross-system RCA benchmark with step-wise causal annotations for LLMs, consisting of 500 instances.
  • Applying PAVE to 11 frontier LLMs shows that recovering the exact root-cause set succeeds in only 20.7% of cases on average.

Key Concepts Mentioned

  • Root cause analysis (RCA)
  • Large language models (LLM)
  • Fault injection
  • Step-wise labeling protocol
  • OpenRCA 2.0

Claims Made by the Document

  • Existing datasets for RCA only label the root cause, not the path connecting it to the observed symptom.
  • PAVE allows for more accurate evaluation of LLMs in RCA by reconstructing causal propagation paths.
  • OpenRCA 2.0 is a new RCA benchmark with step-wise causal annotations for LLMs.

Evidence Used

  • Researchers introduce PAVE...
  • OpenRCA 2.0 is introduced...
  • Recovering the exact root-cause set succeeds in only 20.7% of cases on average.

Methods / Approach

  • PAVE: a step-wise labeling protocol using known interventions from fault injection to reconstruct causal propagation paths for evaluating LLMs in RCA

Limitations Stated by the Authors

The difficulty in accurately identifying the root cause lies somewhere, but the researchers do not specify where exactly.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The paper does not provide information on the specific LLMs used or the methodology of fault injection.

Open Questions

  • Where does the difficulty in accurately identifying the root cause lie?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The researchers do not specify what they mean by 'frontier LLMs'.
  • It is unclear how the 20.7% success rate was calculated, as it is mentioned without any context or supporting information.

Raw Abstract / Excerpt

Root cause analysis (RCA) poses a holistic test of LLM agentic capabilities, such as long-context understanding, multi-step reasoning, and tool use. However, existing datasets suffer from a fundamental gap: they label only the root cause, not the propagation path connecting it to the observed symptom, which largely simplifies the task to naive pattern matching. To support rigorous evaluation, we introduce PAVE, a step-wise labeling protocol that leverages known interventions from fault injection to reconstruct causal propagation paths. The mechanism is forward verification: reasoning from cause to effect rather than inferring backward from symptoms. Applying PAVE yields OpenRCA 2.0 (500 instances), the first cross-system RCA benchmark with step-wise causal annotations for LLM agents. Across 11 frontier LLMs, recovering the exact root-cause set succeeds in only 20.7% of cases on average. To locate where this difficulty lies, we relax the criterion and find what we call the ungrounded diagnosis: agents identify at least one correct root-cause service in 76.0% of cases, but ground that service in a verified causal propagation path to the observed symptom in only 61.5%. Outcome-only evaluation hides this failure mode; step-wise causal ground truth is the missing piece for trustworthy LLM-based RCA agents.