MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Yikun Fu, Bowen Fu, Zhenyu Wu, Shuang Cheng, Xiaowei Sun, Bowen Yang, Zehao Li, Yibo Zhao, Zichen Ding, Zhoumianze Liu, Shijie Wang, Biqing Qi, Bowen Zhou
• Venue: arXiv: cs.AI
• Published: 2026-06-21
• Collected: 2026-06-22
• URL: http://arxiv.org/abs/2606.22557v1
• PDF: https://arxiv.org/pdf/2606.22557v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers have developed a new benchmark for AI agents on macOS desktops called MacAgentBench. The benchmark includes 676 tasks across 25 applications and measures the performance of various models using a deterministic rule-based evaluation method. They found that the best configuration, Claude Opus 4.6 on OpenClaw, achieved a Pass@1 score of 73.7%.

What the Document Says

• The research presents MacAgentBench, a comprehensive benchmark for AI agents on macOS desktops.
• The benchmark includes 676 tasks across 25 applications.
• Nearly 60% of the tasks involve both GUI and CLI interaction.
• MacAgentBench adopts deterministic rule-based evaluation.
• It introduces fine-grained multi-checkpoint scoring with capability annotations for multi-application tasks.
• Experiments were conducted across three frameworks and 16 models.
• The best configuration, Claude Opus 4.6 on OpenClaw, attained a Pass@1 score of 73.7%.

Key Concepts Mentioned

• AI agents
• macOS desktop
• benchmark
• tasks
• applications
• deterministic rule-based evaluation
• fine-grained multi-checkpoint scoring
• Claude Opus 4.6
• OpenClaw

Claims Made by the Document

• Existing benchmarks for macOS fail to capture the capabilities of modern CUAs.
• The best configuration on MacAgentBench achieved a Pass@1 score of 73.7%.

Evidence Used

• Computer use agents (CUAs) have advanced rapidly in desktop automation.
• Existing benchmarks evaluate agents without framework augmentation and rely on binary evaluation.
• MacAgentBench comprises 676 tasks across 25 applications with nearly 60% involving both GUI and CLI interaction.
• The benchmark adopts deterministic rule-based evaluation and introduces fine-grained multi-checkpoint scoring.
• Experiments were conducted across three frameworks and 16 models.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

• The authors do not specify any limitations in the provided text.

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The agent may not be able to determine the specific methodology used for the experiments without additional context or information.

Open Questions

• What are the specific differences between the three frameworks used in the experiments?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The research does not specify which three frameworks were used in the experiments.
• It is unclear how the 'Pass@1' metric is calculated.

Raw Abstract / Excerpt

Computer use agents (CUAs) have advanced rapidly in desktop automation, and a growing number of users deploy CUAs such as OpenClaw on Mac Mini for always-on automation. However, existing benchmarks, including those for macOS, evaluate agents without framework augmentation and rely on binary evaluation. As a result, they fail to capture both the framework capabilities leveraged by modern CUAs and the partial progress on long-horizon, multi-application tasks. We present MacAgentBench, a comprehensive macOS agent benchmark comprising 676 tasks across 25 applications, with nearly 60% involving both GUI and CLI interaction. The benchmark adopts deterministic rule-based evaluation and introduces fine-grained multi-checkpoint scoring with capability annotations for multi-application tasks. Experiments across three frameworks and 16 models show that the best configuration, Claude Opus 4.6 on OpenClaw, attains 73.7% Pass@1, while this advantage is primarily driven by the skill library rather than by framework design. Fine-grained metrics further reveal that models with similar Pass@1 can differ substantially in sub-goal completion. Our code and data are publicly available at https://github.com/JetAstra/MacAgentBench.