OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

Source

Source name: arXiv
Source kind: paper
Authors / organization: Chengyu Shen, Yujie Fu, Gangtao Xin, Yanheng Hou, Wenlong Fei, Guojie Zhu, Jiawei Li, Hongcheng Gao, Runming He, Zhen Hao Wong, Meiyi Qiang, Hao Liang, Zhao Cao, Hao Jiang, Chong Chen, Wentao Zhang
Venue: arXiv: cs.CL
Published: 2026-07-16
Collected: 2026-07-17
URL: http://arxiv.org/abs/2607.14989v1
PDF: https://arxiv.org/pdf/2607.14989v1
DOI: UNKNOWN

Plain-Language Brief

A research paper titled 'OmniaBench' introduces a new benchmark for evaluating the capabilities of general AI agents across various scenarios, using diverse tasks drawn from app stores, product documents, industry resources, and more. The benchmark includes a hierarchical taxonomy of 90 level-1 and 354 level-2 domains, with an accompanying dataset containing 1,431 tasks. The paper reports that even advanced models like Claude-Sonnet-5 and GPT-5.6-Sol struggle to achieve high scores on this benchmark, highlighting areas of improvement for future AI development.

What the Document Says

The paper introduces OmniaBench, a benchmark for evaluating general AI agents
OmniaBench covers diverse scenarios with explicit state spaces
Scenario knowledge is derived from various sources including app stores and human refinement
A hierarchical taxonomy of 90 level-1 and 354 level-2 domains is formed
OmniaBench includes executable environments and tasks through four routes: DAG, DAG-S, Solver, and Program
The benchmark introduces a ten-dimensional capability taxonomy and eight compositional atomic difficulty factors for fine-grained evaluation
The dataset contains 1,431 tasks, with a challenging subset of 644 tasks
Current frontier models struggle on OmniaBench, with Claude-Sonnet-5 and GPT-5.6-Sol achieving Overall Pass@1 scores of only 58.54 and 57.14, respectively
Further analyses reveal differences across domains and capabilities, as well as limitations in planning, constraint maintenance, and adaptive correction
OmniaBench aims to characterize the capability boundaries of general agents

Key Concepts Mentioned

OmniaBench
general AI agents
benchmark
diverse scenarios
explicit state spaces
scenario knowledge
hierarchical taxonomy
app stores
product documents
industry resources
Web retrieval
human refinement
executable environments
single-turn tasks
multi-turn tasks
DAG
DAG-S
Solver
Program
ten-dimensional capability taxonomy
compositional atomic difficulty factors
dataset
Overall Pass@1 scores
Claude-Sonnet-5
GPT-5.6-Sol
planning
constraint maintenance
adaptive correction

Claims Made by the Document

OmniaBench is a benchmark for evaluating general AI agents across diverse scenarios with explicit state spaces

Evidence Used

The paper describes the construction of OmniaBench and its focus on diverse scenarios with explicit state spaces

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large language models are increasingly evolving from text generators into general agents capable of understanding user requests, invoking external tools, and completing complex tasks through interaction. However, existing agent benchmarks often focus on limited scenarios, tool ecosystems, or interaction formats, making it difficult to systematically characterize model capabilities across heterogeneous application settings. We introduce OmniaBench, a benchmark for evaluating general agents across diverse scenarios with explicit state spaces. We derive application-oriented scenario knowledge from app stores, product documents, industry resources, Web retrieval, and human refinement, forming a hierarchical taxonomy that spans ToC, ToB and ToE with 90 level-1 and 354 level-2 domains. Based on this taxonomy, we construct executable environments and synthesize single-turn and multi-turn tasks through four complementary routes: DAG, DAG-S, Solver, and Program. OmniaBench further introduces a ten-dimensional capability taxonomy and eight compositional atomic difficulty factors to support fine-grained evaluation and analysis. The resulting dataset contains 1,431 tasks, together with a challenging subset of 644 tasks designed to reduce evaluation cost and mitigate potential contamination of the full set after public release. The bench presents substantial challenges to current frontier models, with even Claude-Sonnet-5 and GPT-5.6-Sol achieving Overall Pass@1 scores of only 58.54 and 57.14, respectively. Further analyses reveal clear differences across domains and capabilities, as well as persistent limitations in planning, constraint maintenance, and adaptive correction. OmniaBench provides a broad and diagnostic benchmark for characterizing the capability boundaries of general agents.