Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Divake Kumar, Sina Tayebati, Devashri Naik, Amanda Sofie Rios, Nilesh Ahuja, Omesh Tickoo, Ranganath Krishnan, Amit Ranjan Trivedi
• Venue: arXiv: cs.LG
• Published: 2026-06-24
• Collected: 2026-06-25
• URL: http://arxiv.org/abs/2606.25760v1
• PDF: https://arxiv.org/pdf/2606.25760v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers have developed a benchmark called Argus to test the reliability of computer-use agents that convert vision-language model predictions into executable clicks. The study compares 27 methods across four vision-language models and four datasets, as well as eight methods for three frontier vendors where some information is unavailable. The findings show that the ranking of these methods is stable within a specific model but can degrade when moving between models or observable interfaces. The study also finds that hidden-state and density methods are the most stable, and specific methods like CoCoA-1MCA, Focus, sampling-based scores, and verbalized self-assessment perform well in certain regimes. The authors suggest that closed-source UQ should be recalibrated on the target rather than extrapolated from other sources.

What the Document Says

• The study presents Argus, a benchmark for post-hoc uncertainty quantification (UQ) in single-step executable GUI grounding.
• It compares 27 methods across four vision-language models and four datasets, and eight methods for three frontier vendors where some information is unavailable.
• The findings indicate that UQ rankings are stable within a fixed model but can degrade across different models and observable interfaces.
• Hidden-state and density methods are found to be the most stable open-weight family, while CoCoA-1MCA, Focus, sampling-based scores, and verbalized self-assessment perform well in specific regimes.
• Within-model ranking transfer is strong, but cross-tier transfer to closed-source vendors is only +0.08, suggesting that closed-source UQ should be reranked on the target rather than extrapolated.
• Conformal click regions show score-level discrimination is not enough for deployment and that locally weighted disks shrink radii when the plug-in UQ is calibrated, but coverage degrades under calibration-test or interface mismatch.
• The authors release per-item records, calibration/test splits, UQ scores, and analysis scripts for regime-aware UQ selection in GUI agents.

Key Concepts Mentioned

• computer-use agent
• vision-language model (VLM)
• GUI grounding
• uncertainty quantification (UQ)
• Argus benchmark
• post-hoc UQ
• logit-based scores
• sampling and consistency measures
• hidden-state estimators
• density estimators
• attention-based scores
• P(True)
• verbalised-confidence prompting
• split-conformal prediction
• Conformal click regions

Claims Made by the Document

• Reliable uncertainty estimates are essential for computer-use agents.
• Evidence on post-hoc UQ is fragmented and it's unclear whether rankings stay stable with changes in agent, benchmark, or observable interface.
• Hidden-state and density methods are the most stable open-weight family.
• CoCoA-1MCA, Focus, sampling-based scores, and verbalized self-assessment perform well in specific regimes.
• Within-model ranking transfer is strong but cross-tier transfer to closed-source vendors is only +0.08.
• Conformal click regions show score-level discrimination is not enough for deployment.

Evidence Used

• The text mentions the importance of reliable uncertainty estimates for computer-use agents multiple times.
• The study compares various methods across different models and datasets, indicating a fragmented evidence base.
• Hidden-state and density methods are mentioned as being the most stable open-weight family.
• Specific methods like CoCoA-1MCA, Focus, sampling-based scores, and verbalized self-assessment are mentioned as performing well in certain regimes.
• The text states that within-model ranking transfer is strong but cross-tier transfer to closed-source vendors is only +0.08.
• Conformal click regions are discussed as showing score-level discrimination is not enough for deployment.

Methods / Approach

• Argus benchmark
• logit-based scores
• sampling and consistency measures
• hidden-state estimators
• density estimators
• attention-based scores
• P(True)
• verbalised-confidence prompting
• split-conformal prediction
• Conformal click regions

Limitations Stated by the Authors

• The study focuses on single-step executable GUI grounding, which may not be representative of more complex scenarios.

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The agent's understanding of the specific technical terms and methodologies used in the research might be limited.
• The agent does not have the ability to perform further analysis or experiments based on the provided data.

Open Questions

• What are the implications of these findings for the development and implementation of computer-use agents?
• How can the stability of UQ rankings be improved across different models and observable interfaces?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The study does not provide details on how the specific methods were implemented or evaluated.
• The text mentions three frontier vendors but does not name them.

Raw Abstract / Excerpt

Computer-use agents turn vision-language model (VLM) predictions into executable GUI clicks, so reliable uncertainty estimates are essential for rejection, calibration, miss-severity ranking, and spatial safety regions. Yet evidence on post-hoc uncertainty quantification (UQ) for these agents is fragmented across isolated model and dataset pairs, leaving it unclear whether UQ rankings stay stable when the agent, benchmark, or observable interface changes. We present Argus, a cross-regime benchmark for post-hoc UQ in single-step executable GUI grounding: a 27-method open-weight matrix over 4 VLM agents and 4 datasets, plus an 8-method closed-source matrix across 3 frontier vendors where logits, hidden states, and attention maps are unavailable. Evaluated methods span logit-based scores, sampling and consistency measures, hidden-state and density estimators (Mahalanobis, SAPLMA), attention-based scores, P(True) and verbalised-confidence prompting, and split-conformal prediction. The main finding is selective transfer: UQ rankings are stable across datasets for a fixed model, but degrade across model classes and observable interfaces. Hidden-state and density methods are the most stable open-weight family, while CoCoA-1MCA, Focus, sampling-based scores, and verbalised self-assessment win in specific regimes. Within-model ranking transfer is strong (Spearman rho up to 0.969), but cross-tier transfer to closed-source vendors averages only +0.08, so closed-source UQ should be reranked on the target rather than extrapolated. Conformal click regions show score-level discrimination is not enough for deployment: locally weighted disks shrink radii by 40-60% when the plug-in UQ is calibrated, but coverage degrades under calibration-test or interface mismatch. We release per-item records, calibration/test splits, UQ scores, and analysis scripts for regime-aware UQ selection in GUI agents.