Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

Source

arXiv

arxiv.org/abs/2607.14989v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Chengyu Shen, Yujie Fu, Gangtao Xin, Yanheng Hou, Wenlong Fei, Guojie Zhu, Jiawei Li, Hongcheng Gao, Runming He, Zhen Hao Wong, Meiyi Qiang, Hao Liang, Zhao Cao, Hao Jiang, Chong Chen, Wentao Zhang
  • Venue: arXiv: cs.CL
  • Published: 2026-07-16
  • Collected: 2026-07-17
  • URL: http://arxiv.org/abs/2607.14989v1
  • PDF: https://arxiv.org/pdf/2607.14989v1
  • DOI: UNKNOWN

Plain-Language Brief

A research paper titled 'OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios' introduces a new benchmark called OmniaBench, designed to evaluate the capabilities of general AI agents in diverse scenarios with explicit state spaces. The benchmark uses a hierarchical taxonomy derived from various sources and includes 1,431 tasks, with 644 challenging tasks to reduce evaluation cost. The results show that current frontier models struggle with the challenges presented by OmniaBench.

What the Document Says

  • The paper introduces a benchmark called OmniaBench for evaluating general AI agents across diverse scenarios.
  • OmniaBench uses a hierarchical taxonomy derived from app stores, product documents, industry resources, web retrieval, and human refinement.
  • The taxonomy spans ToC, ToB, and ToE with 90 level-1 domains and 354 level-2 domains.
  • OmniaBench constructs executable environments and synthesizes single-turn and multi-turn tasks through four routes: DAG, DAG-S, Solver, and Program.
  • The benchmark includes a ten-dimensional capability taxonomy and eight compositional atomic difficulty factors for fine-grained evaluation.
  • The dataset contains 1,431 tasks with a challenging subset of 644 tasks.
  • The results show that even Claude-Sonnet-5 and GPT-5.6-Sol achieve Overall Pass@1 scores of only 58.54 and 57.14, respectively.
  • Further analyses reveal differences across domains and capabilities, as well as limitations in planning, constraint maintenance, and adaptive correction.

Key Concepts Mentioned

  • OmniaBench
  • general AI agents
  • diverse scenarios
  • hierarchical taxonomy
  • executable environments
  • single-turn tasks
  • multi-turn tasks
  • DAG
  • DAG-S
  • Solver
  • Program
  • capability taxonomy
  • atomic difficulty factors
  • challenging subset
  • Claude-Sonnet-5
  • GPT-5.6-Sol
  • Overall Pass@1 scores
  • planning
  • constraint maintenance
  • adaptive correction

Claims Made by the Document

  • OmniaBench is a useful benchmark for evaluating general AI agents across diverse scenarios.
  • The hierarchical taxonomy used by OmniaBench covers ToC, ToB, and ToE with 90 level-1 domains and 354 level-2 domains.
  • OmniaBench constructs executable environments and synthesizes single-turn and multi-turn tasks through four complementary routes.

Evidence Used

  • The paper describes the creation of OmniaBench for evaluating general AI agents across diverse scenarios.
  • The text details the derivation of a hierarchical taxonomy used by OmniaBench.
  • The paper explains how OmniaBench constructs executable environments and synthesizes tasks.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large language models are increasingly evolving from text generators into general agents capable of understanding user requests, invoking external tools, and completing complex tasks through interaction. However, existing agent benchmarks often focus on limited scenarios, tool ecosystems, or interaction formats, making it difficult to systematically characterize model capabilities across heterogeneous application settings. We introduce OmniaBench, a benchmark for evaluating general agents across diverse scenarios with explicit state spaces. We derive application-oriented scenario knowledge from app stores, product documents, industry resources, Web retrieval, and human refinement, forming a hierarchical taxonomy that spans ToC, ToB and ToE with 90 level-1 and 354 level-2 domains. Based on this taxonomy, we construct executable environments and synthesize single-turn and multi-turn tasks through four complementary routes: DAG, DAG-S, Solver, and Program. OmniaBench further introduces a ten-dimensional capability taxonomy and eight compositional atomic difficulty factors to support fine-grained evaluation and analysis. The resulting dataset contains 1,431 tasks, together with a challenging subset of 644 tasks designed to reduce evaluation cost and mitigate potential contamination of the full set after public release. The bench presents substantial challenges to current frontier models, with even Claude-Sonnet-5 and GPT-5.6-Sol achieving Overall Pass@1 scores of only 58.54 and 57.14, respectively. Further analyses reveal clear differences across domains and capabilities, as well as persistent limitations in planning, constraint maintenance, and adaptive correction. OmniaBench provides a broad and diagnostic benchmark for characterizing the capability boundaries of general agents.