ORCA-bench: How Ready Are Language Model Agents for Oncall?

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Albert Gong, Kyuseong Choi, Abhineet Agarwal, Jason Schechner, Ryan Huang, Raj Agrawal, Anish Agarwal, Raaz Dwivedi
• Venue: arXiv: cs.CL
• Published: 2026-07-30
• Collected: 2026-07-31
• URL: http://arxiv.org/abs/2607.28545v1
• PDF: https://arxiv.org/pdf/2607.28545v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers introduce ORCA-bench, a new benchmark that tests the ability of large language models to perform root cause analysis (RCA) in a production-like setting. The benchmark includes a microservice system with six days of metrics, logs, traces, and source code, along with 1079 RCA tasks. The best performing model achieved an accuracy of 25.3% on medium difficulty tasks and 10.0% on hard tasks. The researchers note that these results are from a controlled test environment and the performance of language models in real-world production systems may require significant engineering investment.

What the Document Says

• ORCA-bench is a new benchmark for testing large language models.
• It includes a microservice system with metrics, logs, traces, and source code for six days.
• There are 1079 RCA tasks included in the benchmark.
• Ground-truth symptoms are verified by expert SREs.
• The best performing model achieved an accuracy of 25.3% on medium difficulty tasks and 10.0% on hard tasks.
• The researchers note that these results are from a controlled test environment.
• The weakest model hallucinates implausible root causes in 40% of incident reports.

Key Concepts Mentioned

• ORCA-bench
• large language models
• root cause analysis (RCA)
• microservice system
• metrics
• logs
• traces
• source code
• SREs

Claims Made by the Document

• The best performing model achieved an accuracy of 25.3% on medium difficulty tasks.
• The best performing model achieved an accuracy of 10.0% on hard tasks.

Evidence Used

• The researchers report the accuracy of the best performing model in the text.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large language models can write, patch, and search code, but oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began. We introduce ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting. ORCA-bench pairs a live OpenTelemetry-instrumented microservice system--exposing six days of metrics, logs, and traces through real telemetry interfaces (Prometheus, Jaeger, and OpenSearch via Grafana) and full source-code access--with 1,079 RCA tasks that systematically vary report specificity, time-to-detection, and co-occurring fault scenarios. Ground-truth symptoms are curated and signed off by expert SREs, and our LLM-as-judge is independently re-scored by humans (Cohen's $κ_w=0.90$). Across five frontier agents, the best RCA Accuracy is 25.3% on Medium-difficulty tasks (the realistic-input setting) and 10.0% on Hard--a gap that remains even with Claude Fable 5. The weakest model hallucinates an implausible root cause in 40% of incident reports, and removing source-code access degrades every metric. Crucially, these are performances on a curated 50 GB / six-day testbed with tasks investigated in isolation on a system whose code and instrumentation are public. Since real production systems are order of magnitudes larger, more dynamic, and more idiosyncratic, the gap we report is a lower bound on the engineering investment required before frontier coding agents can be safely entrusted with production reliability. We release the public set at https://hub.harborframework.com/datasets/orca-bench/ORCA-bench.