OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Qiushi Sun, Kanzhi Cheng, Yian Wang, Bowen Yang, Hang Yan, Liheng Chen, Fangzhi Xu, Zichen Ding, Nuo Chen, Jialin Cao, Xingdong Gong, Zehao Li, Kaiming Jin, Xinfeng Yuan, Zhoumianze Liu, Jingyang Gong, Zhangyue Yin, Jiahui Gao, Zhiyong Wu, Tianbao Xie, Jianbing Zhang, Ben Kao, Lingpeng Kong
• Venue: arXiv: cs.AI
• Published: 2026-07-30
• Collected: 2026-07-31
• URL: http://arxiv.org/abs/2607.28609v1
• PDF: https://arxiv.org/pdf/2607.28609v1
• DOI: UNKNOWN

Plain-Language Brief

The research introduces OSReward, a new benchmark for evaluating the reliability of vision-language models (VLMs) used to judge computer-using agents' trajectories. The study finds that state-of-the-art VLMs have a leniency bias, mislabeling failed runs as successes, and there is a gap between reliable but expensive models and affordable open models. To address this gap, the researchers introduce OS-Shepherd, an open reward model that provides low-cost, stable, and reliable reward signals, matching commercial judges at 30-60% lower cost.

What the Document Says

• The paper presents OSReward, a benchmark for evaluating the reliability of vision-language models (VLMs) in judging computer-using agents' trajectories.
• The study finds that even state-of-the-art VLMs have a systematic leniency bias, mislabeling failed runs as successes.
• Reliable but expensive models exist, while affordable open models trail behind.
• To address this gap, the researchers introduce OS-Shepherd, an open reward model that provides low-cost, stable, and reliable reward signals.

Key Concepts Mentioned

• Computer-using agents
• Vision-language models
• CUA trajectories
• Evaluation
• Reinforcement learning
• Data curation
• OSReward
• OSReward-Hard
• OSReward-Multi
• Leniency bias
• OS-Shepherd

Claims Made by the Document

• State-of-the-art VLMs have a leniency bias.
• Reliable but expensive models exist, while affordable open models trail behind.

Evidence Used

• The study's findings based on the evaluation of various VLM judges.

Methods / Approach

• Introduction and development of OSReward benchmark, creation of OSReward-Hard and OSReward-Multi sets, extensive analyses

Limitations Stated by the Authors

• The study does not provide details on the specific methods used in the evaluation of VLM judges.

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The text lacks information on the specific CUA backbones and human-verified instructions used in the study.
• The text does not specify the exact cost comparison between OS-Shepherd and commercial judges.

Open Questions

• What specific VLM models were evaluated in the study?
• What is the exact cost comparison between OS-Shepherd and commercial judges?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The paper does not mention which specific vision-language models were used in the evaluation.
• The paper presents a cost comparison but lacks details on the exact costs involved.

Raw Abstract / Excerpt

Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's actions, states, and reasoning. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms, then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge set concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. The few reliable enough to trust are too expensive to run at scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. On it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60% lower cost than the frontier. Extensive analyses further inform the design of reliable CUA reward at scale. Our code, benchmark, dataset, and model checkpoints are available at https://os-copilot.github.io/OSReward-Home/.