Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Plover: Steering GUI Agents through Plan-Centric Interaction

Source

arXiv

arxiv.org/abs/2607.15193v1

Read the source

Source

Plain-Language Brief

The paper presents Plover, a system for controlling graphical user interface (GUI) automation agents in real-world environments. It makes planning and adaptation visible to users, allowing them to supervise, correct, or inspect the agent's behavior.

What the Document Says

  • Plover is a system for GUI automation designed to handle dynamic layouts, unexpected dialogs, and evolving interface states in real-world environments.
  • The system operates on screenshots and natural language instructions.
  • Plover externalizes task plans and replanning as persistent, inspectable, and revisable artifacts.
  • It has a planner--executor architecture that supports explicit supervision, localized correction, natural-language guidance, and screenshot-grounded interventions.
  • The system preserves prior progress during repair.
  • A study with six participants was conducted to inform the interaction design of Plover.
  • Evaluation through benchmark failure-case repair and scenario-based workflow analyses showed that many GUI automation failures can be repaired when plans remain visible and interventions are localized, and that explicit replanning improves transparency, control, and adaptability.

Key Concepts Mentioned

  • GUI automation
  • plan-centric system
  • planner--executor architecture
  • task plans
  • replanning
  • screenshots
  • natural language instructions

Claims Made by the Document

  • Plover improves flexibility in GUI automation.
  • Externalizing task plans and replanning aids users in supervising, correcting, or inspecting the agent's behavior.
  • The system preserves prior progress during repair.
  • Many autonomous GUI-agent failures are structurally repairable when plans remain visible and interventions are localized.
  • Explicit replanning helps make GUI automation more transparent, controllable, and adaptable.

Evidence Used

  • The text describes the design and functioning of Plover as a system for GUI automation.
  • The evaluation results show improvements in transparency, control, and adaptability due to explicit replanning.

Methods / Approach

  • A study with six participants was conducted to inform the interaction design.
  • Evaluation through benchmark failure-case repair and scenario-based workflow analyses.

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

The agent may not have access to the specific details of the study or evaluation methods used in the paper.

Open Questions

  • What were the specific details of the study and evaluation methods used?
  • How does the system perform compared to other GUI automation systems?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The text suggests that the study involved six participants, but no further details are provided about the nature or scope of the study.

Raw Abstract / Excerpt

Graphical user interface (GUI) automation remains challenging in real-world environments, where dynamic layouts, unexpected dialogs, and evolving interface states can cause autonomous agents to drift from user intent. Recent vision-based multimodal agents improve flexibility by operating directly over screenshots and natural language instructions, but planning and adaptation often remain internal, limiting users' ability to inspect, supervise, or correct system behavior. We present Plover, a plan-centric vision-based GUI automation system that externalizes task plans and replanning as persistent, inspectable, and revisable artifacts. Through a planner--executor architecture, Plover supports explicit supervision of evolving execution, localized correction through editable plans, natural-language guidance, and screenshot-grounded interventions, while preserving prior progress during repair. A formative study with six participants informed the interaction design. We then evaluate Plover through benchmark failure-case repair and scenario-based workflow analyses. Our results show that many autonomous GUI-agent failures are structurally repairable when plans remain visible and interventions are localized, and that explicit replanning helps make GUI automation more transparent, controllable, and adaptable.