Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

ORCA-bench: How Ready Are Language Model Agents for Oncall?

Source

arXiv

arxiv.org/abs/2607.28545v1

Read the source

Source

Plain-Language Brief

The study presents ORCA-bench, a benchmark for evaluating the performance of large language models in oncall root cause analysis (RCA). The benchmark exposes a microservice system with six days of data and 1,079 RCA tasks. The best-performing model achieved an accuracy of 25.3% on medium difficulty tasks and 10.0% on hard tasks, indicating significant room for improvement. The authors note that the reported performance is a lower bound for real-world systems due to their size and idiosyncrasies.

What the Document Says

  • The study introduces ORCA-bench, a benchmark for evaluating general-purpose coding agents in production-fidelity oncall settings.
  • ORCA-bench consists of a live OpenTelemetry-instrumented microservice system with six days of metrics, logs, traces, and source code access.
  • The benchmark includes 1,079 RCA tasks that vary report specificity, time-to-detection, and co-occurring fault scenarios.
  • Ground-truth symptoms are curated by expert SREs and independently re-scored by humans with Cohen's kappa score of 0.90.
  • The best RCA accuracy across five frontier agents is 25.3% on medium difficulty tasks and 10.0% on hard tasks.
  • The weakest model hallucinates an implausible root cause in 40% of incident reports.
  • Removing source-code access degrades every metric.
  • The performance reported is a lower bound for real-world systems due to their size, dynamics, and idiosyncrasies.
  • The public set of the benchmark is released at https://hub.harborframework.com/datasets/orca-bench/ORCA-bench.

Key Concepts Mentioned

  • Large language models
  • Oncall root cause analysis (RCA)
  • ORCA-bench
  • OpenTelemetry-instrumented microservice system
  • 1,079 RCA tasks
  • Ground-truth symptoms
  • Cohen's kappa score
  • Frontier coding agents
  • Accuracy
  • Implausible root cause
  • Source-code access
  • Production reliability

Claims Made by the Document

  • The best RCA accuracy for frontier coding agents is low on medium and hard difficulty tasks.
  • Removing source-code access degrades performance.

Evidence Used

  • The reported best RCA Accuracy is 25.3% on Medium-difficulty tasks and 10.0% on Hard.
  • Removing source-code access degrades every metric as stated in the text.

Methods / Approach

  • Creating a benchmark (ORCA-bench) for evaluating general-purpose coding agents in production-fidelity oncall settings
  • Using expert SREs to curate ground-truth symptoms and human re-scoring
  • Measuring RCA Accuracy of frontier coding agents

Limitations Stated by the Authors

  • The reported performance is a lower bound for real-world systems due to their size, dynamics, and idiosyncrasies.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The study does not discuss potential limitations or biases in the curation of ground-truth symptoms by expert SREs.
  • The study does not provide information about the diversity or representativeness of the incident reports used in the evaluation.

Open Questions

  • What are the specific characteristics of real-world systems that make them more challenging for frontier coding agents than the benchmarked system?
  • How do different models and approaches perform on ORCA-bench, and what factors contribute to their performance?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The study mentions five frontier agents, but it is unclear which specific models or systems these refer to.

Raw Abstract / Excerpt

Large language models can write, patch, and search code, but oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began. We introduce ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting. ORCA-bench pairs a live OpenTelemetry-instrumented microservice system--exposing six days of metrics, logs, and traces through real telemetry interfaces (Prometheus, Jaeger, and OpenSearch via Grafana) and full source-code access--with 1,079 RCA tasks that systematically vary report specificity, time-to-detection, and co-occurring fault scenarios. Ground-truth symptoms are curated and signed off by expert SREs, and our LLM-as-judge is independently re-scored by humans (Cohen's $κ_w=0.90$). Across five frontier agents, the best RCA Accuracy is 25.3% on Medium-difficulty tasks (the realistic-input setting) and 10.0% on Hard--a gap that remains even with Claude Fable 5. The weakest model hallucinates an implausible root cause in 40% of incident reports, and removing source-code access degrades every metric. Crucially, these are performances on a curated 50 GB / six-day testbed with tasks investigated in isolation on a system whose code and instrumentation are public. Since real production systems are order of magnitudes larger, more dynamic, and more idiosyncratic, the gap we report is a lower bound on the engineering investment required before frontier coding agents can be safely entrusted with production reliability. We release the public set at https://hub.harborframework.com/datasets/orca-bench/ORCA-bench.