Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Shiyu Xuan, Zechao Li
• Venue: arXiv: cs.CV
• Published: 2026-08-11
• Collected: 2026-08-12
• URL: http://arxiv.org/abs/2608.11191v1
• PDF: https://arxiv.org/pdf/2608.11191v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

The paper proposes a Test-Time Self-Evolving framework that allows GUI agents to adapt and improve after deployment without requiring human-annotated ground truth data. The framework includes four components: exploration, evaluation, reflection, and internalization. The agent first explores unseen interfaces, then evaluates its explorations with a Reflector which provides reasoning reflections. A Reflection-Guided On-Policy Self-Distillation method is used to internalize the reflection knowledge into the model weights. A Contrastive Calibration method is also designed to prevent errors during failed explorations. The framework demonstrated effectiveness on six benchmarks, achieving an average accuracy improvement of 7.4% over the base model.

What the Document Says

• The paper introduces a Test-Time Self-Evolving framework for GUI agents that adapt after deployment without human-annotated ground truth data.
• The framework includes exploration, evaluation, reflection, and internalization components.
• During exploration, the agent predicts grounding coordinates for given instructions on unseen interfaces.
• An MLLM-based Reflector is used to evaluate the generated results and provide reasoning reflections.
• Reflection-Guided On-Policy Self-Distillation is proposed to internalize reflection knowledge into the model weights.
• A Contrastive Calibration method is designed to prevent errors during failed explorations.
• The framework showed effectiveness on six benchmarks, improving average accuracy by 7.4% over the base model.
• This is reportedly the first work to successfully exploit on-policy self-distillation for test-time adaptation in GUI visual grounding.

Key Concepts Mentioned

• GUI agents
• Test-Time Self-Evolving framework
• Exploration
• Evaluation
• Reflection
• Internalization
• MLLM-based Reflector
• Reflection-Guided On-Policy Self-Distillation
• Contrastive Calibration

Claims Made by the Document

• The proposed framework allows GUI agents to adapt and improve after deployment without human-annotated ground truth data.
• The framework showed effectiveness on six benchmarks, improving average accuracy by 7.4% over the base model.

Evidence Used

• The text describes a Test-Time Self-Evolving framework that enables models to improve after deployment without human-annotated ground truth.
• The text presents experimental results demonstrating the framework's effectiveness on six benchmarks, achieving an average accuracy improvement of 7.4% over the base model.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

GUI Visual Grounding is a fundamental capability for GUI agents. Existing models typically freeze their parameters after deployment, limiting their ability to adapt to unseen interfaces. Although recent methods attempt to adapt models via test-time reinforcement learning, they cannot reflect upon failed exploration. To overcome this, we propose a Test-Time Self-Evolving framework that enables models to improve after deployment without human-annotated ground truth. It constructs a closed-loop of Exploration, Evaluation, Reflection, and Internalization. Specifically, the agent first explores unseen interfaces by predicting grounding coordinates for given instructions. To evaluate these explorations, we introduce an MLLM-based Reflector to assess the generated results and provide the corresponding reasoning reflections. To internalize reflection knowledge into the model weights, we propose Reflection-Guided On-Policy Self-Distillation, which translates high-level reasoning into dense token-level supervision via a conditioned self-teacher. Furthermore, we design a Contrastive Calibration method to prevent incorrect auto-regressive prefixes from corrupting the supervisory signals during failed explorations. Extensive experiments across six benchmarks demonstrate our framework's effectiveness, achieving an average accuracy improvement of 7.4% over the base model. To the best of our knowledge, this is the first work to successfully exploit on-policy self-distillation for test-time adaptation in GUI visual grounding. By filling the gap in post-deployment adaptation, our framework completes the self-evolving capability of GUI agents. The code will be released.