Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching

Source

arXiv

arxiv.org/abs/2608.09654v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Yuke Li, Xuehan Hou
  • Venue: arXiv: cs.AI
  • Published: 2026-08-10
  • Collected: 2026-08-11
  • URL: http://arxiv.org/abs/2608.09654v1
  • PDF: https://arxiv.org/pdf/2608.09654v1
  • DOI: UNKNOWN
  • Registry group: academic_apis_and_preprints
  • Evidence role: canonical_academic_index
  • Language: en
  • Country / region: UNKNOWN

Plain-Language Brief

Researchers propose a new framework for grounding user instructions in graphical user interfaces (GUI). The framework uses a frozen multimodal large language model (MLLM) to parse instructions and a dedicated grounding model to locate the precise elements without learning coordinate regression. This approach improves the accuracy of GUI grounding by over 20% on ScreenSpot-Pro and increases success rate and element selection rate on Mind2Web, compared to end-to-end systems.

What the Document Says

  • GUI agents are using MLLMs that operate directly on screenshots.
  • The core task is GUI grounding, which translates abstract user instructions into precise element coordinates.
  • Conventional grounding models lack semantic richness for interpreting abstract instructions.
  • End-to-end MLLMs suffer from coordinate hallucinations due to deficient fine-grained perception.
  • The proposed framework decouples instruction understanding from layout-aware localization.
  • A frozen MLLM elaborates the abstract instruction into a structured visual description rich in layout cues.
  • The grounding model performs regression-free localization by matching against layout-prior candidates.
  • The grounding model is trained with only Text/Icon binary labels, not requiring coordinate regression parameters.

Key Concepts Mentioned

  • GUI
  • Graphical User Interface
  • MLLMs
  • GUI grounding
  • Abstract user instructions
  • Precise element coordinates
  • Semantic richness
  • Coordinate hallucinations
  • Fine-grained perception
  • Layout cues
  • Regression-free localization
  • Layout-prior candidates
  • Text/Icon binary labels
  • End-to-end systems
  • ScreenSpot-Pro
  • Mind2Web

Claims Made by the Document

  • Our method achieves over 20% improvement in grounding accuracy on ScreenSpot-Pro.
  • On Mind2Web, our method raises success rate and element selection rate by more than 15%.

Evidence Used

  • On ScreenSpot-Pro, our method improves grounding accuracy over end-to-end systems.
  • On Mind2Web, it increases success rate and element selection rate by more than 15%.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

GUI agents are shifting from metadata-dependent large language models to purely visual multimodal large language models (MLLMs) that operate directly on screenshots. The core task, GUI grounding, requires translating abstract user instructions into precise element coordinates. This task faces a persistent dual obstacle: conventional grounding models lack the semantic richness to interpret abstract instructions, while end-to-end MLLMs suffer from coordinate hallucinations caused by deficient fine-grained perception. We propose a regression-free framework where a frozen MLLM performs instruction parsing and a dedicated grounding model handles precise localization without learning any coordinate regression. A frozen MLLM first elaborates the abstract instruction into a structured visual description rich in layout cues. These descriptions are then fed to a novel Layout-Aware GUI Grounding Model, which performs regression-free localization by matching against layout-prior candidates, inherently suppressing hallucinations and avoiding expensive fine-tuning. The grounding model is trained with only Text/Icon binary labels, requiring no coordinate regression parameters. On ScreenSpot-Pro, our method achieves over 20% improvement in grounding accuracy over end-to-end systems; on Mind2Web, it raises success rate and element selection rate by more than 15%. These results demonstrate that decoupling instruction understanding from layout-aware localization effectively resolves the core challenges of GUI interaction.