OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Chengyu Shen, Yujie Fu, Gangtao Xin, Yanheng Hou, Wenlong Fei, Guojie Zhu, Jiawei Li, Hongcheng Gao, Runming He, Zhen Hao Wong, Meiyi Qiang, Hao Liang, Zhao Cao, Hao Jiang, Chong Chen, Wentao Zhang
• Venue: arXiv: cs.CL
• Published: 2026-07-16
• Collected: 2026-07-17
• URL: http://arxiv.org/abs/2607.14989v1
• PDF: https://arxiv.org/pdf/2607.14989v1
• DOI: UNKNOWN

Plain-Language Brief

OmniaBench is a new benchmark for evaluating the capabilities of general AI agents across diverse scenarios. It includes over 1,400 tasks with a hierarchical taxonomy of 90 level-1 and 354 level-2 domains based on app stores, product documents, industry resources, web retrieval, and human refinement. The benchmark challenges current state-of-the-art models like Claude-Sonnet-5 and GPT-5.6-Sol with scores of only 58.54 and 57.14 respectively. It also highlights limitations in planning, constraint maintenance, and adaptive correction.

What the Document Says

• OmniaBench is a benchmark for evaluating general AI agents across diverse scenarios.
• It uses a hierarchical taxonomy derived from various sources to categorize the tasks.
• The taxonomy consists of 90 level-1 and 354 level-2 domains.
• OmniaBench includes 1,431 tasks with four routes for task synthesis: DAG, DAG-S, Solver, and Program.
• It introduces a ten-dimensional capability taxonomy and eight compositional atomic difficulty factors for fine-grained evaluation.
• The dataset also contains a challenging subset of 644 tasks to reduce evaluation cost and mitigate potential contamination.
• Current frontier models like Claude-Sonnet-5 and GPT-5.6-Sol have Overall Pass@1 scores of only 58.54 and 57.14 respectively on this benchmark.
• Further analyses reveal differences across domains and capabilities, as well as limitations in planning, constraint maintenance, and adaptive correction.

Key Concepts Mentioned

• OmniaBench
• general AI agents
• diverse scenarios
• hierarchical taxonomy
• application-oriented scenario knowledge
• executable environments
• single-turn and multi-turn tasks
• DAG, DAG-S, Solver, Program
• capability taxonomy
• atomic difficulty factors
• Overall Pass@1 scores
• Claude-Sonnet-5
• GPT-5.6-Sol
• planning
• constraint maintenance
• adaptive correction

Claims Made by the Document

• OmniaBench is a benchmark for evaluating general AI agents across diverse scenarios.
• It provides a challenging evaluation for current frontier models like Claude-Sonnet-5 and GPT-5.6-Sol.
• It highlights limitations in planning, constraint maintenance, and adaptive correction.

Evidence Used

• The text describes the introduction of OmniaBench as a benchmark for evaluating general AI agents across diverse scenarios.
• The text states that current frontier models like Claude-Sonnet-5 and GPT-5.6-Sol have low Overall Pass@1 scores on this benchmark.
• The text mentions the limitations in planning, constraint maintenance, and adaptive correction revealed through further analyses.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

• The paper does not provide detailed information on how the tasks in OmniaBench were designed and validated.
• The paper only provides a brief overview of the methods used for task synthesis, but the specific details are not included in the abstract.

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The abstract does not provide enough information to fully understand how OmniaBench was constructed or validated.

Open Questions

• What is the purpose of OmniaBench and how does it differ from other benchmarks?
• How were the tasks in OmniaBench designed and validated?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The paper does not explicitly state the purpose of OmniaBench, but it is presented as a diagnostic benchmark for characterizing the capability boundaries of general agents.
• The tasks in OmniaBench are synthesized through four complementary routes: DAG, DAG-S, Solver, and Program, but the specific methods used are not detailed in the abstract.

Raw Abstract / Excerpt

Large language models are increasingly evolving from text generators into general agents capable of understanding user requests, invoking external tools, and completing complex tasks through interaction. However, existing agent benchmarks often focus on limited scenarios, tool ecosystems, or interaction formats, making it difficult to systematically characterize model capabilities across heterogeneous application settings. We introduce OmniaBench, a benchmark for evaluating general agents across diverse scenarios with explicit state spaces. We derive application-oriented scenario knowledge from app stores, product documents, industry resources, Web retrieval, and human refinement, forming a hierarchical taxonomy that spans ToC, ToB and ToE with 90 level-1 and 354 level-2 domains. Based on this taxonomy, we construct executable environments and synthesize single-turn and multi-turn tasks through four complementary routes: DAG, DAG-S, Solver, and Program. OmniaBench further introduces a ten-dimensional capability taxonomy and eight compositional atomic difficulty factors to support fine-grained evaluation and analysis. The resulting dataset contains 1,431 tasks, together with a challenging subset of 644 tasks designed to reduce evaluation cost and mitigate potential contamination of the full set after public release. The bench presents substantial challenges to current frontier models, with even Claude-Sonnet-5 and GPT-5.6-Sol achieving Overall Pass@1 scores of only 58.54 and 57.14, respectively. Further analyses reveal clear differences across domains and capabilities, as well as persistent limitations in planning, constraint maintenance, and adaptive correction. OmniaBench provides a broad and diagnostic benchmark for characterizing the capability boundaries of general agents.