computer-use agents
6 notes, newest first.
How Benchmarks Mis-Score Computer-Use Agents
A research paper discusses issues with the scoring system of computer-use agents (software that navigates the web and operates desktop software). The authors argue that current scores are often inaccurate due to…
How Benchmarks Mis-Score Computer-Use Agents
The paper discusses issues with the scoring of Computer-Use Agents (CUAs) using brittle scripted oracles, highlighting problems such as outdated tasks, omitted decisive visual evidence, and hidden failure causes in…
How Benchmarks Mis-Score Computer-Use Agents
The paper discusses issues with the benchmark scores of Computer-Use Agents (CUA), highlighting that the scores are often inaccurate due to brittle scripted oracles. The authors propose a reliability framework and…
Rethinking Inference-Time Scaling in Local Computer-Use Agents: Failure Modes and Compute Tradeoffs
The paper presents an empirical study on the effectiveness of inference-time scaling in locally deployed autonomous computer-use agents (CUAs) under strict hardware constraints. The research focuses on four dimensions…
Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets
Researchers have developed a benchmark called Argus to test the reliability of computer-use agents that convert vision-language model predictions into executable clicks. The study compares 27 methods across four…
Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity?
Researchers present AgentCIBench, a tool for evaluating the privacy risks of computer-use agents (CUAs) that access personal data across various applications. The study finds that most frontier CUAs have a high failure…
