Benchmarking LLM Judges for Mobile Agent Evaluation

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Ziqiang Wan, Li Gu, Zhixiang Chi, Zhi Liu, Seyed Mehdi Ayyoubzadeh, Yuanhao Yu, Yang Wang
• Venue: arXiv preprint
• Published: 2026-08-12
• Collected: 2026-08-14
• URL: https://arxiv.org/abs/2608.11434
• PDF: https://arxiv.org/pdf/2608.11434
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

A benchmark study of LLM judges used to evaluate mobile-agent trajectories.

What the Document Says

The paper introduces MobileJudgeBench, a benchmark for LLM-based judges of mobile-agent trajectories. It contains 931 human-annotated trajectories spanning six mobile-agent benchmarks, four agent models, and 68 apps. The authors report that a simple sampled-screenshot judge can be competitive with purpose-built pipelines, that the backbone model is a major determinant of judge utility, and that different backbones show conservative or permissive failure profiles.

Key Concepts Mentioned

• mobile agent evaluation
• LLM judges
• benchmarking
• human annotation

Claims Made by the Document

• The benchmark contains 931 annotated trajectories across six benchmarks, four agent models, and 68 apps.
• A simple screenshot-sampling baseline is competitive with some purpose-built judges.
• Judge backbones are associated with different failure profiles.

Evidence Used

• Human-annotated trajectories and comparative evaluation of judge methods, as described in the abstract.

Methods / Approach

The source describes a benchmark and comparative evaluation of six judge methods across mobile-agent trajectories.

Limitations Stated by the Authors

The abstract does not state the full benchmark construction protocol or limits to generalization beyond the evaluated apps, agents, and judge backbones.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The note is based on the arXiv abstract and search record, not the full paper.

Open Questions

• How do results change across unseen apps and mobile operating-system versions?
• What annotation agreement and calibration procedures were used?

Notes for Future Q&A

May include interpretation. Labeled where it does.

Use this note for questions about mobile-agent evaluation and LLM-as-judge reliability.

Raw Abstract / Excerpt

The paper introduces MobileJudgeBench, a benchmark for LLM-based judges of mobile-agent trajectories. It contains 931 human-annotated trajectories spanning six mobile-agent benchmarks, four agent models, and 68 apps. The authors report that a simple sampled-screenshot judge can be competitive with purpose-built pipelines, that the backbone model is a major determinant of judge utility, and that different backbones show conservative or permissive failure profiles.