Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

Source

arXiv

arxiv.org/abs/2607.28609v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Qiushi Sun, Kanzhi Cheng, Yian Wang, Bowen Yang, Hang Yan, Liheng Chen, Fangzhi Xu, Zichen Ding, Nuo Chen, Jialin Cao, Xingdong Gong, Zehao Li, Kaiming Jin, Xinfeng Yuan, Zhoumianze Liu, Jingyang Gong, Zhangyue Yin, Jiahui Gao, Zhiyong Wu, Tianbao Xie, Jianbing Zhang, Ben Kao, Lingpeng Kong
  • Venue: arXiv: cs.AI
  • Published: 2026-07-30
  • Collected: 2026-07-31
  • URL: http://arxiv.org/abs/2607.28609v1
  • PDF: https://arxiv.org/pdf/2607.28609v1
  • DOI: UNKNOWN

Plain-Language Brief

The research introduces OSReward, a new benchmark for evaluating the reliability of vision-language models (VLMs) used to judge computer-using agents (CUAs). The study finds that even state-of-the-art VLMs have a systematic leniency bias, mislabeling failed runs as successes. To address this issue, the researchers develop OS-Shepherd, an open reward model that provides low-cost, stable, and reliable reward signals, performing as well as commercial judges at 30-60% lower cost.

What the Document Says

  • Computer-using agents (CUAs) are being evaluated by vision-language models (VLMs).
  • A new benchmark called OSReward is introduced to evaluate VLMs on CUA trajectories.
  • The OSReward benchmark includes diverse agent backbones executing human-verified instructions across platforms, with ground-truth verdicts from multi-stage human annotation.
  • OSReward-Hard focuses on challenging cases and OSReward-Multi on fine-grained efficiency and alignment scoring.
  • The study finds that state-of-the-art VLMs have a systematic leniency bias, mislabeling failed runs as successes.
  • Affordable open models trail behind the reliable enough to trust for large-scale use.
  • OS-Shepherd (9B and 35B), an open reward model, is developed to provide low-cost, stable, and reliable reward signals, performing as well as commercial judges at 30-60% lower cost.
  • The researchers' code, benchmark, dataset, and model checkpoints are available.

Key Concepts Mentioned

  • Computer-using agents (CUAs)
  • Vision-language models (VLMs)
  • OSReward
  • OS-Shepherd

Claims Made by the Document

  • State-of-the-art VLMs have a systematic leniency bias.
  • Affordable open models trail behind the reliable enough to trust for large-scale use.

Evidence Used

  • The study findings based on OSReward evaluation of VLM judges.
  • Comparative performance data between state-of-the-art VLMs, affordable open models, and OS-Shepherd.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

  • The authors do not explicitly address the causes of the leniency bias in state-of-the-art VLMs.
  • Affordable open models are shown to trail behind reliable models, but no specific strategies for improvement are provided.

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

  • What are the specific causes of the leniency bias in state-of-the-art VLMs?
  • How can affordable open models be improved to match the performance of commercial judges?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The text does not explicitly mention how the authors address the causes of leniency bias in state-of-the-art VLMs.
  • Affordable open models are shown to trail behind reliable models, but the study does not provide specific strategies for improving their performance.

Raw Abstract / Excerpt

Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's actions, states, and reasoning. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms, then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge set concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. The few reliable enough to trust are too expensive to run at scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. On it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60% lower cost than the frontier. Extensive analyses further inform the design of reliable CUA reward at scale. Our code, benchmark, dataset, and model checkpoints are available at https://os-copilot.github.io/OSReward-Home/.