Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Divake Kumar, Sina Tayebati, Devashri Naik, Amanda Sofie Rios, Nilesh Ahuja, Omesh Tickoo, Ranganath Krishnan, Amit Ranjan Trivedi
• Venue: arXiv: cs.LG
• Published: 2026-06-24
• Collected: 2026-06-25
• URL: http://arxiv.org/abs/2606.25760v1
• PDF: https://arxiv.org/pdf/2606.25760v1
• DOI: UNKNOWN

Plain-Language Brief

The study introduces Argus, a benchmark for post-hoc uncertainty quantification (UQ) in single-step executable GUI grounding across various vision-language models and datasets. The research compares 27 methods of UQ across 4 VLM agents and 4 datasets, as well as an 8-method comparison among 3 frontier vendors with limited data availability. The key finding is that UQ rankings are consistent within datasets but vary across models and observable interfaces. Hidden-state and density methods are the most stable, while specific methods excel in certain regimes. The study also finds that score-level discrimination is not sufficient for deployment, and locally weighted disks shrink radii significantly when UQ is calibrated but coverage degrades under calibration testing or interface mismatch.

What the Document Says

• The paper presents Argus, a benchmark for post-hoc uncertainty quantification (UQ) in single-step executable GUI grounding.
• It compares 27 methods of UQ across 4 VLM agents and 4 datasets, and an additional 8-method comparison among 3 frontier vendors with limited data availability.
• Hidden-state and density methods are the most stable open-weight family in the study.
• CoCoA-1MCA, Focus, sampling-based scores, and verbalised self-assessment win in specific regimes.
• Within-model ranking transfer is strong, but cross-tier transfer to closed-source vendors averages only +0.08, suggesting that closed-source UQ should be reranked on the target rather than extrapolated.

Key Concepts Mentioned

• Uncertainty Quantification
• Computer-use Agents
• Vision-Language Models (VLMs)
• GUI Grounding Datasets
• Argus Benchmark

Claims Made by the Document

• Reliable uncertainty estimates are essential for various tasks in computer-use agents.
• Evidence on post-hoc UQ is fragmented and unclear about its stability across changes.
• Selective transfer: UQ rankings are stable across datasets for a fixed model, but degrade across model classes and observable interfaces.

Evidence Used

• The paper presents a benchmark and compares various methods of UQ across multiple agents and datasets.
• The study finds that hidden-state and density methods are the most stable open-weight family, while specific methods excel in certain regimes.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Computer-use agents turn vision-language model (VLM) predictions into executable GUI clicks, so reliable uncertainty estimates are essential for rejection, calibration, miss-severity ranking, and spatial safety regions. Yet evidence on post-hoc uncertainty quantification (UQ) for these agents is fragmented across isolated model and dataset pairs, leaving it unclear whether UQ rankings stay stable when the agent, benchmark, or observable interface changes. We present Argus, a cross-regime benchmark for post-hoc UQ in single-step executable GUI grounding: a 27-method open-weight matrix over 4 VLM agents and 4 datasets, plus an 8-method closed-source matrix across 3 frontier vendors where logits, hidden states, and attention maps are unavailable. Evaluated methods span logit-based scores, sampling and consistency measures, hidden-state and density estimators (Mahalanobis, SAPLMA), attention-based scores, P(True) and verbalised-confidence prompting, and split-conformal prediction. The main finding is selective transfer: UQ rankings are stable across datasets for a fixed model, but degrade across model classes and observable interfaces. Hidden-state and density methods are the most stable open-weight family, while CoCoA-1MCA, Focus, sampling-based scores, and verbalised self-assessment win in specific regimes. Within-model ranking transfer is strong (Spearman rho up to 0.969), but cross-tier transfer to closed-source vendors averages only +0.08, so closed-source UQ should be reranked on the target rather than extrapolated. Conformal click regions show score-level discrimination is not enough for deployment: locally weighted disks shrink radii by 40-60% when the plug-in UQ is calibrated, but coverage degrades under calibration-test or interface mismatch. We release per-item records, calibration/test splits, UQ scores, and analysis scripts for regime-aware UQ selection in GUI agents.