Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

Source

arXiv

arxiv.org/abs/2607.28609v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Qiushi Sun, Kanzhi Cheng, Yian Wang, Bowen Yang, Hang Yan, Liheng Chen, Fangzhi Xu, Zichen Ding, Nuo Chen, Jialin Cao, Xingdong Gong, Zehao Li, Kaiming Jin, Xinfeng Yuan, Zhoumianze Liu, Jingyang Gong, Zhangyue Yin, Jiahui Gao, Zhiyong Wu, Tianbao Xie, Jianbing Zhang, Ben Kao, Lingpeng Kong
  • Venue: arXiv: cs.AI
  • Published: 2026-07-30
  • Collected: 2026-07-31
  • URL: http://arxiv.org/abs/2607.28609v1
  • PDF: https://arxiv.org/pdf/2607.28609v1
  • DOI: UNKNOWN

Plain-Language Brief

The paper introduces OSReward, a benchmark for evaluating the reliability of vision-language models (VLMs) used as judges for computer-using agents (CUAs). The benchmark consists of CUA trajectories from various agent backbones executing human-verified instructions across platforms and rigorously labeled with ground-truth verdicts. The study finds that even state-of-the-art VLM models have a systematic leniency bias, mislabeling failed runs as successes. To address this gap, the authors introduce OS-Shepherd, an open reward model that provides low-cost and reliable reward signals, matching commercial judges at 30-60% lower cost.

What the Document Says

  • The paper introduces OSReward, a benchmark for evaluating the reliability of vision-language models (VLMs) used as judges for computer-using agents (CUAs).
  • The benchmark consists of CUA trajectories from various agent backbones executing human-verified instructions across platforms and rigorously labeled with ground-truth verdicts.
  • State-of-the-art VLM models have a systematic leniency bias, mislabeling failed runs as successes.
  • To address this gap, the authors introduce OS-Shepherd, an open reward model.

Key Concepts Mentioned

  • Computer-using agents
  • Vision-language models
  • CUA trajectories
  • reliability evaluation
  • OSReward
  • OS-Shepherd

Claims Made by the Document

  • State-of-the-art VLM models have a systematic leniency bias.
  • OS-Shepherd provides low-cost and reliable reward signals, matching commercial judges at 30-60% lower cost.

Evidence Used

  • The study finds that even state-of-the-art VLM models have a systematic leniency bias.
  • To close this gap, we construct and release OS-Shepherd (9B and 35B), open reward models

Methods / Approach

  • Introduction of OSReward benchmark
  • Multi-stage human annotation
  • Training of OS-Shepherd models

Limitations Stated by the Authors

The study does not provide specific information about the agent backbones or the platforms used in the OSReward benchmark.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The text does not specify how the low-cost and stability of OS-Shepherd were quantified or compared to commercial judges.

Open Questions

  • What are the specific agent backbones and platforms used in the OSReward benchmark?
  • How was the cost and stability of OS-Shepherd quantified and compared to commercial judges?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper does not provide details about the agent backbones or platforms used in the OSReward benchmark. This information may be essential for understanding the applicability and validity of the study's findings.
  • The text does not specify how the low-cost and stability of OS-Shepherd were quantified or compared to commercial judges. More details about this comparison could help in assessing the practical significance of the findings.

Raw Abstract / Excerpt

Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's actions, states, and reasoning. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms, then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge set concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. The few reliable enough to trust are too expensive to run at scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. On it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60% lower cost than the frontier. Extensive analyses further inform the design of reliable CUA reward at scale. Our code, benchmark, dataset, and model checkpoints are available at https://os-copilot.github.io/OSReward-Home/.