Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Shiyu Xuan, Zechao Li
• Venue: arXiv: cs.CV
• Published: 2026-08-11
• Collected: 2026-08-12
• URL: http://arxiv.org/abs/2608.11191v1
• PDF: https://arxiv.org/pdf/2608.11191v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

The authors propose a new framework called Test-Time Self-Evolving for GUI (Graphical User Interface) Visual Grounding. This method allows models to improve after deployment without the need for human-annotated ground truth. The framework consists of Exploration, Evaluation, Reflection, and Internalization stages. It uses a self-teacher to translate high-level reasoning into token-level supervision and a Contrastive Calibration method to prevent incorrect signals during failed explorations.

What the Document Says

• The paper discusses a new framework for GUI Visual Grounding called Test-Time Self-Evolving.
• The framework enables models to improve after deployment without human-annotated ground truth.
• It consists of Exploration, Evaluation, Reflection, and Internalization stages.
• A self-teacher is used to translate high-level reasoning into token-level supervision.
• A Contrastive Calibration method is used to prevent incorrect signals during failed explorations.
• The framework has been tested across six benchmarks and showed an average accuracy improvement of 7.4% over the base model.

Key Concepts Mentioned

• GUI Visual Grounding
• Test-Time Self-Evolving framework
• Exploration
• Evaluation
• Reflection
• Internalization
• self-teacher
• Contrastive Calibration

Claims Made by the Document

• The proposed Test-Time Self-Evolving framework enables models to improve after deployment without human-annotated ground truth.
• The framework achieves an average accuracy improvement of 7.4% over the base model.

Evidence Used

• The authors explain the different stages of their proposed framework and how it allows for post-deployment adaptation.
• They provide results from extensive experiments across six benchmarks demonstrating the framework's effectiveness.

Methods / Approach

• Reflection-Guided On-Policy Self-Distillation
• MLLM-based Reflector

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

GUI Visual Grounding is a fundamental capability for GUI agents. Existing models typically freeze their parameters after deployment, limiting their ability to adapt to unseen interfaces. Although recent methods attempt to adapt models via test-time reinforcement learning, they cannot reflect upon failed exploration. To overcome this, we propose a Test-Time Self-Evolving framework that enables models to improve after deployment without human-annotated ground truth. It constructs a closed-loop of Exploration, Evaluation, Reflection, and Internalization. Specifically, the agent first explores unseen interfaces by predicting grounding coordinates for given instructions. To evaluate these explorations, we introduce an MLLM-based Reflector to assess the generated results and provide the corresponding reasoning reflections. To internalize reflection knowledge into the model weights, we propose Reflection-Guided On-Policy Self-Distillation, which translates high-level reasoning into dense token-level supervision via a conditioned self-teacher. Furthermore, we design a Contrastive Calibration method to prevent incorrect auto-regressive prefixes from corrupting the supervisory signals during failed explorations. Extensive experiments across six benchmarks demonstrate our framework's effectiveness, achieving an average accuracy improvement of 7.4% over the base model. To the best of our knowledge, this is the first work to successfully exploit on-policy self-distillation for test-time adaptation in GUI visual grounding. By filling the gap in post-deployment adaptation, our framework completes the self-evolving capability of GUI agents. The code will be released.