Plover: Steering GUI Agents through Plan-Centric Interaction

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Madhumitha Venkatesan, Shicheng Wen, Jiajing Guo, Jorge Piazentin Ono, Liu Ren, Dongyu Liu
• Venue: arXiv: cs.AI
• Published: 2026-07-16
• Collected: 2026-07-17
• URL: http://arxiv.org/abs/2607.15193v1
• PDF: https://arxiv.org/pdf/2607.15193v1
• DOI: UNKNOWN

Plain-Language Brief

The paper introduces Plover, a plan-centric GUI automation system designed to overcome challenges in real-world environments. It allows users to supervise, correct, and inspect the system's behavior by externalizing task plans and replanning as persistent, editable artifacts.

What the Document Says

• The paper discusses a system called Plover for graphical user interface (GUI) automation.
• Plover operates in real-world environments with dynamic layouts, unexpected dialogs, and evolving interface states.
• The system improves flexibility by operating directly over screenshots and natural language instructions.
• Planning and adaptation are externalized as persistent, inspectable, and revisable artifacts.
• Plover has a planner--executor architecture.
• It supports explicit supervision of evolving execution, localized correction through editable plans, natural-language guidance, and screenshot-grounded interventions.
• Prior progress is preserved during repair in Plover.
• The interaction design was informed by a formative study with six participants.

Key Concepts Mentioned

• GUI automation
• plan-centric
• planner--executor architecture
• dynamic layouts
• unexpected dialogs

Claims Made by the Document

• Plover overcomes challenges in real-world GUI automation.
• Externalizing planning and adaptation helps users supervise, correct, and inspect the system's behavior.

Evidence Used

• The paper describes Plover as a solution for GUI automation challenges.
• The text mentions that plans are externalized in Plover.

Methods / Approach

• A planner--executor architecture is used.
• Formative study with six participants informs interaction design.

Limitations Stated by the Authors

Not stated in source.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The evaluation of the system was not described in detail.

Open Questions

• What specific methods does Plover use for plan-centric GUI automation?
• How is prior progress preserved during repair in Plover?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper provides a brief overview of the system and its features, but does not go into detail about how it achieves plan-centric GUI automation or how prior progress is preserved during repair.

Raw Abstract / Excerpt

Graphical user interface (GUI) automation remains challenging in real-world environments, where dynamic layouts, unexpected dialogs, and evolving interface states can cause autonomous agents to drift from user intent. Recent vision-based multimodal agents improve flexibility by operating directly over screenshots and natural language instructions, but planning and adaptation often remain internal, limiting users' ability to inspect, supervise, or correct system behavior. We present Plover, a plan-centric vision-based GUI automation system that externalizes task plans and replanning as persistent, inspectable, and revisable artifacts. Through a planner--executor architecture, Plover supports explicit supervision of evolving execution, localized correction through editable plans, natural-language guidance, and screenshot-grounded interventions, while preserving prior progress during repair. A formative study with six participants informed the interaction design. We then evaluate Plover through benchmark failure-case repair and scenario-based workflow analyses. Our results show that many autonomous GUI-agent failures are structurally repairable when plans remain visible and interventions are localized, and that explicit replanning helps make GUI automation more transparent, controllable, and adaptable.