Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding

Source

arXiv

arxiv.org/abs/2607.27155v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Jingbo Zhou, Yusai Zhao, Qi Bao, Jingjia Cao, Zhenghai Chen, Chang Gao, Kaiqi Guo, Muxin Guo, Mingxuan Li, Xinjiang Lu, Yanru Ma, Yixiong Xiao, Zenghui Zhang, Le Zhang, Hua Wu
  • Venue: arXiv: cs.AI
  • Published: 2026-07-29
  • Collected: 2026-07-30
  • URL: http://arxiv.org/abs/2607.27155v1
  • PDF: https://arxiv.org/pdf/2607.27155v1
  • DOI: UNKNOWN

Plain-Language Brief

Researchers have developed a new benchmark called OmegaUse-OfficeVal to test the performance of large language model (LLM) agents in completing long-horizon office-suite tasks. The benchmark includes 100 tasks that require an average of 2.32 hours of human labor and are paired with economic signals for comparison between human costs and LLM inference costs. The researchers evaluate several frontier LLMs against a human baseline but find that while the LLMs are faster and cheaper, they have not yet reached human-level deliverable quality.

What the Document Says

  • The text is about a new benchmark for evaluating large language model (LLM) agents called OmegaUse-OfficeVal.
  • The benchmark consists of 100 long-horizon office-suite tasks derived from practitioners' requests.
  • Each task requires an average of 2.32 hours of human labor to complete.
  • The tasks are paired with two economic signals: human labor time and a task price proxy for comparison between human costs and LLM inference costs.
  • The benchmark supports stable evaluation through code-based verifiers from fine-grained rubrics.
  • Several frontier LLMs were evaluated along with a human baseline.
  • While all evaluated LLMs are cheaper and faster than human workers, they have not yet reached human-level deliverable quality.

Key Concepts Mentioned

  • large language model (LLM) agents
  • OmegaUse-OfficeVal benchmark
  • long-horizon office-suite tasks
  • economic grounding
  • human labor time
  • task price proxy

Claims Made by the Document

  • The OmegaUse-OfficeVal benchmark is for evaluating LLM agents on long-horizon office-suite tasks.
  • The benchmark consists of 100 tasks that require an average of 2.32 hours of human labor.
  • Each task is paired with two economic signals: human labor time and a task price proxy.
  • The benchmark supports stable evaluation through code-based verifiers.
  • Several frontier LLMs were evaluated along with a human baseline.
  • While all evaluated LLMs are cheaper and faster than human workers, they have not yet reached human-level deliverable quality.

Evidence Used

  • The text describes the new benchmark and its components

Methods / Approach

  • The use of code-based verifiers from fine-grained rubrics for stable evaluation

Limitations Stated by the Authors

  • The LLMs have not yet reached human-level deliverable quality.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The data provided only shows the results of one study, and it's unknown how generalizable these findings are to other studies or situations.

Open Questions

  • How can the quality of LLMs be improved to reach human-level deliverable quality?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The text does not provide information on the specific tasks included in the benchmark, or the exact methodology used for the evaluation.

Raw Abstract / Excerpt

Large language model (LLM) agents are increasingly expected to assist users in completing tasks. However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding. The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and dataset are fully open-sourced, and more information is available on our project website: https://omegause-officeval.github.io.