How Benchmarks Mis-Score Computer-Use Agents

Source

Source name: arXiv
Source kind: paper
Authors / organization: Zihan Dong, Zhiyuan Ma, Zekun Wang, Yunqing Li, Zirou Liu, Ruixuan Deng, Qishi Zhan, Rui Qian
Venue: arXiv: cs.AI
Published: 2026-07-30
Collected: 2026-07-31
URL: http://arxiv.org/abs/2607.28367v1
PDF: https://arxiv.org/pdf/2607.28367v1
DOI: UNKNOWN

Plain-Language Brief

The paper discusses issues with the benchmark scores of Computer-Use Agents (CUA), highlighting that the scores are often inaccurate due to brittle scripted oracles. The authors propose a reliability framework and analyze 150 public failure-scored trajectories from various benchmarks, finding that 15.3% of FAIL verdicts were incorrect and suggest design rules for CUA evaluation.

What the Document Says

The text discusses Computer-Use Agents (CUA) that browse the web and operate desktop software.
It mentions a problem with benchmark scores, which are output of pipelines where tasks can be stale, trajectories may omit crucial visual evidence, evaluators might reject valid alternatives, and aggregate reports could hide the cause of failure.
The authors propose a reliability framework divided into task construction, trajectory observation, scoring, and reporting.
They audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks.
They find that 15.3% of FAIL verdicts were incorrect: 10.7% are evaluator false negatives and 4.7% are broken tasks.
For genuine failures, a three-tier diagnostic taxonomy is presented showing that verification/feedback and planning failures dominate execution/grounding errors.
The authors connect these findings to newer long-horizon CUA benchmarks and derive stage-specific design rules for CUA evaluation.

Key Concepts Mentioned

Computer-Use Agents (CUA)
benchmark scores
brittle scripted oracles
reliability framework
task construction
trajectory observation
scoring
reporting
failure-scored trajectories
web benchmarks
enterprise-workflow benchmarks
desktop-control benchmarks
long-horizon CUA benchmarks
design rules

Claims Made by the Document

Benchmark scores for Computer-Use Agents are often inaccurate.
15.3% of FAIL verdicts are incorrect: 10.7% are evaluator false negatives and 4.7% are broken tasks.
Verification/feedback and planning failures dominate execution/grounding errors.

Evidence Used

The text describes the issues with benchmark scores for Computer-Use Agents.
The authors present their findings after auditing 150 public failure-scored trajectories.
They provide a three-tier diagnostic taxonomy to support their claim about the types of failures.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Computer-use agents (CUA) are being deployed to browse the web and operate desktop software, yet their benchmark scores are still commonly produced by brittle scripted oracles. A score is the output of a pipeline in which tasks can be stale, trajectories can omit decisive visual evidence, evaluators can reject valid alternatives, and aggregate reports can hide the cause of failure. We organize these problems into a reliability framework spanning task construction, trajectory observation, scoring, and reporting. We then audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3% of FAIL verdicts are wrong: 10.7% are evaluator false negatives and 4.7% are broken tasks. For genuine failures, a three-tier diagnostic taxonomy shows that verification/feedback and planning failures dominate execution/grounding errors, while a single scalar success rate can not explain. We connect these findings to newer long-horizon CUA benchmarks and derive stage-specific design rules for CUA evaluation.