Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

ORCA-bench: How Ready Are Language Model Agents for Oncall?

Source

arXiv

arxiv.org/abs/2607.28545v1

Read the source

Source

Plain-Language Brief

The paper introduces ORCA-bench, a benchmark for testing the performance of large language models in oncall root cause analysis (RCA). The benchmark includes a live microservice system with six days of metrics, logs, and traces, and 1079 RCA tasks that vary in report specificity, time-to-detection, and co-occurring fault scenarios. The best RCA accuracy for frontier agents on medium difficulty tasks is 25.3%, and only 10% on hard tasks. Hallucinations of implausible root causes are common among the weakest models, and removing source-code access degrades all metrics. The authors suggest that the reported performance gap is a lower bound for the engineering investment required before frontier coding agents can be trusted with production reliability.

What the Document Says

  • The paper presents ORCA-bench, a benchmark for testing general-purpose coding agents in a production-like oncall setting.
  • ORCA-bench includes a live OpenTelemetry-instrumented microservice system with six days of metrics, logs, and traces.
  • It also includes 1079 RCA tasks that vary in report specificity, time-to-detection, and co-occurring fault scenarios.
  • Ground-truth symptoms are curated by expert SREs and independently re-scored by humans.
  • The best RCA Accuracy for frontier agents on Medium-difficulty tasks is 25.3%.
  • On Hard tasks, the best RCA accuracy is 10.0%.
  • Hallucinations of implausible root causes are common among the weakest models.
  • Removing source-code access degrades all metrics.
  • The reported performance gap is a lower bound on the engineering investment required before frontier coding agents can be trusted with production reliability.
  • The authors release the public set at https://hub.harborframework.com/datasets/orca-bench/ORCA-bench.

Key Concepts Mentioned

  • Large language models
  • Oncall root cause analysis (RCA)
  • ORCA-bench
  • OpenTelemetry-instrumented microservice system
  • Metrics, logs, and traces
  • Real telemetry interfaces (Prometheus, Jaeger, OpenSearch via Grafana)
  • Full source-code access
  • RCA tasks
  • Ground-truth symptoms
  • Expert SREs
  • Human re-scoring
  • Cohen's $κ_w
  • Frontier agents
  • RCA Accuracy
  • Implausible root causes
  • Source-code access
  • Production reliability

Claims Made by the Document

  • Large language models are not ready for oncall RCA as they perform poorly on a benchmark that simulates production-like settings.

Evidence Used

  • The reported RCA Accuracy for frontier agents is low, especially on hard tasks.
  • Hallucinations of implausible root causes are common among the weakest models.
  • Removing source-code access degrades all metrics.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large language models can write, patch, and search code, but oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began. We introduce ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting. ORCA-bench pairs a live OpenTelemetry-instrumented microservice system--exposing six days of metrics, logs, and traces through real telemetry interfaces (Prometheus, Jaeger, and OpenSearch via Grafana) and full source-code access--with 1,079 RCA tasks that systematically vary report specificity, time-to-detection, and co-occurring fault scenarios. Ground-truth symptoms are curated and signed off by expert SREs, and our LLM-as-judge is independently re-scored by humans (Cohen's $κ_w=0.90$). Across five frontier agents, the best RCA Accuracy is 25.3% on Medium-difficulty tasks (the realistic-input setting) and 10.0% on Hard--a gap that remains even with Claude Fable 5. The weakest model hallucinates an implausible root cause in 40% of incident reports, and removing source-code access degrades every metric. Crucially, these are performances on a curated 50 GB / six-day testbed with tasks investigated in isolation on a system whose code and instrumentation are public. Since real production systems are order of magnitudes larger, more dynamic, and more idiosyncratic, the gap we report is a lower bound on the engineering investment required before frontier coding agents can be safely entrusted with production reliability. We release the public set at https://hub.harborframework.com/datasets/orca-bench/ORCA-bench.