How Benchmarks Mis-Score Computer-Use Agents

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Zihan Dong, Zhiyuan Ma, Zekun Wang, Yunqing Li, Zirou Liu, Ruixuan Deng, Qishi Zhan, Rui Qian
• Venue: arXiv: cs.AI
• Published: 2026-07-30
• Collected: 2026-07-31
• URL: http://arxiv.org/abs/2607.28367v1
• PDF: https://arxiv.org/pdf/2607.28367v1
• DOI: UNKNOWN

Plain-Language Brief

A research paper discusses issues with the scoring system of computer-use agents (software that navigates the web and operates desktop software). The authors argue that current scores are often inaccurate due to several problems such as outdated tasks, omitted crucial evidence, incorrect evaluations, and hidden failure causes. The paper proposes a reliability framework for task construction, observation, scoring, and reporting, and finds that 15.3% of reported failures are erroneous. The authors also suggest design rules for computer-use agent evaluation based on their findings.

What the Document Says

• Computer-use agents (CUAs) are software that navigate the web and operate desktop software.
• The scores of CUAs are often produced by brittle scripted oracles.
• The paper identifies problems in scores: tasks can be stale, trajectories may omit crucial visual evidence, evaluators can reject valid alternatives, and aggregate reports can hide failure causes.
• The authors propose a reliability framework for task construction, observation, scoring, and reporting.
• 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks were audited.
• 15.3% of FAIL verdicts were found to be wrong: 10.7% were evaluator false negatives and 4.7% were broken tasks.
• For genuine failures, a three-tier diagnostic taxonomy shows that verification/feedback and planning failures dominate execution/grounding errors.
• A single scalar success rate cannot explain the findings.

Key Concepts Mentioned

• Computer-use agents
• benchmarks
• scores
• reliability framework
• failure verdicts
• evaluator false negatives
• broken tasks
• diagnostic taxonomy

Claims Made by the Document

• Current scores of CUAs are often inaccurate.
• 15.3% of reported failures are erroneous.
• Verification/feedback and planning failures dominate execution/grounding errors.

Evidence Used

• The paper discusses the problems in current scores.
• The audit of public failure-scored trajectories.
• The three-tier diagnostic taxonomy findings.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Computer-use agents (CUA) are being deployed to browse the web and operate desktop software, yet their benchmark scores are still commonly produced by brittle scripted oracles. A score is the output of a pipeline in which tasks can be stale, trajectories can omit decisive visual evidence, evaluators can reject valid alternatives, and aggregate reports can hide the cause of failure. We organize these problems into a reliability framework spanning task construction, trajectory observation, scoring, and reporting. We then audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3% of FAIL verdicts are wrong: 10.7% are evaluator false negatives and 4.7% are broken tasks. For genuine failures, a three-tier diagnostic taxonomy shows that verification/feedback and planning failures dominate execution/grounding errors, while a single scalar success rate can not explain. We connect these findings to newer long-horizon CUA benchmarks and derive stage-specific design rules for CUA evaluation.