Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation

Source

arXiv

arxiv.org/abs/2606.24515v1

Read the source

Source

Plain-Language Brief

Researchers propose a framework for training Computer-Use Agents (CUAs) using reinforcement learning and autonomous vision-language evaluation. This method provides a scalable supervision signal, improving success rates in GUI environments compared to zero-shot baselines and raw evaluator rewards.

What the Document Says

  • Computer-Use Agents (CUAs) are designed to perform high-level user goals within graphical user interfaces.
  • The paper discusses the challenge of reinforcement learning for CUAs due to the lack of scalable, machine-readable reward signals in open-ended desktop environments.
  • The authors propose a framework that uses autonomous vision-language evaluation as a supervision signal for GUI agents during policy optimization.
  • An autonomous evaluator judges task completion based on a final screenshot and the original instruction without requiring task-specific heuristics or manual labels.
  • Because autonomous evaluators are imperfect, their feedback is modeled as a noisy binary reward channel.
  • A noise-corrected reward estimator for Proximal Policy Optimization (PPO) is derived to address the noise.
  • The experiments show that corrected evaluator rewards outperform both zero-shot baselines and raw evaluator rewards, improving success rates by an average of 12.6 percentage points over zero-shot performance and 5.1 points over raw evaluator fine-tuning.
  • The results indicate that autonomous evaluation can be a practical reward signal for RL in GUI environments when evaluator noise is explicitly modeled and corrected.

Key Concepts Mentioned

  • Computer-Use Agents
  • Reinforcement Learning
  • GUI Environments
  • Vision-Language Model
  • Autonomous Evaluation
  • Proximal Policy Optimization

Claims Made by the Document

  • The proposed framework can improve success rates for Computer-Use Agents in GUI environments.
  • Autonomous evaluators are imperfect and provide noisy feedback.
  • Noise-corrected reward estimator for PPO improves the performance of the framework.

Evidence Used

  • Experiments across macOSWorld, Windows Agent Arena, and OSWorld show that corrected evaluator rewards outperform baselines and raw evaluator rewards.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Computer-Use Agents (CUAs) execute high-level user goals by perceiving and acting directly within graphical user interfaces. However, reinforcement learning for CUAs remains difficult because open-ended desktop environments rarely provide scalable, machine-readable reward signals: task success is often visually grounded and hard to specify with handcrafted reward functions or dense manual labels. We propose an RL fine-tuning framework that uses autonomous vision-language evaluation as a scalable supervision signal for GUI agents. Given a final screenshot and the original instruction, a Vision-Language Model judges task completion and provides terminal feedback without task-specific heuristics or manual labels during policy optimization. Because autonomous evaluators are imperfect, we model their feedback as a noisy binary reward channel and derive a noise-corrected reward estimator for Proximal Policy Optimization. Experiments across macOSWorld, Windows Agent Arena, and OSWorld show that corrected evaluator rewards outperform both zero-shot baselines and raw evaluator rewards, improving success rates by an average of 12.6 percentage points over zero-shot performance and 5.1 points over raw evaluator fine-tuning. These results suggest that autonomous evaluation can serve as a practical reward signal for RL in GUI environments when evaluator noise is explicitly modeled and corrected.