OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Qiushi Sun, Kanzhi Cheng, Yian Wang, Bowen Yang, Hang Yan, Liheng Chen, Fangzhi Xu, Zichen Ding, Nuo Chen, Jialin Cao, Xingdong Gong, Zehao Li, Kaiming Jin, Xinfeng Yuan, Zhoumianze Liu, Jingyang Gong, Zhangyue Yin, Jiahui Gao, Zhiyong Wu, Tianbao Xie, Jianbing Zhang, Ben Kao, Lingpeng Kong
• Venue: arXiv: cs.AI
• Published: 2026-07-30
• Collected: 2026-07-31
• URL: http://arxiv.org/abs/2607.28609v1
• PDF: https://arxiv.org/pdf/2607.28609v1
• DOI: UNKNOWN

Plain-Language Brief

The research introduces OSReward, a new benchmark for evaluating the reliability of Vision-Language Models (VLMs) that judge Computer-Using Agents' (CUAs) trajectories. The study finds that even state-of-the-art VLMs have a systematic leniency bias and are either too expensive or not reliable enough to handle large-scale CUA evaluations. To address this issue, the researchers developed OS-Shepherd, an open-source reward model that provides low-cost, stable, and reliable rewards signals at 30-60% lower cost than current industry standards.

What the Document Says

• The study introduces OSReward, a benchmark for evaluating VLMs judging CUAs' trajectories.
• OSReward includes diverse agent backbones executing human-verified instructions across platforms.
• Ground-truth verdicts are obtained through multi-stage human annotation.
• The evaluation finds that state-of-the-art VLMs have a leniency bias, mislabeling failed runs as successes.
• Reliable VLMs are too expensive for large-scale evaluations.
• Open models trail behind in reliability.
• To close the gap, OS-Shepherd (9B and 35B), open reward models were developed.
• OS-Shepherd provides low-cost, stable, and reliable reward signals at 30-60% lower cost than the frontier.

Key Concepts Mentioned

• Computer-Using Agents (CUAs)
• Vision-Language Models (VLMs)
• OSReward
• OS-Shepherd
• reliability

Claims Made by the Document

• State-of-the-art VLMs have a systematic leniency bias.
• Reliable VLMs are too expensive for large-scale evaluations.
• Open models trail behind in reliability compared to commercial judges.

Evidence Used

• The text states that state-of-the-art VLMs mislabel failed runs as successes.
• It is mentioned that reliable VLMs are too expensive for large-scale evaluations.
• The text indicates that open models trail behind in reliability compared to commercial judges.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's actions, states, and reasoning. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms, then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge set concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. The few reliable enough to trust are too expensive to run at scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. On it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60% lower cost than the frontier. Extensive analyses further inform the design of reliable CUA reward at scale. Our code, benchmark, dataset, and model checkpoints are available at https://os-copilot.github.io/OSReward-Home/.