Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Plover: Steering GUI Agents through Plan-Centric Interaction

Source

arXiv

arxiv.org/abs/2607.15193v1

Read the source

Source

Plain-Language Brief

The paper introduces Plover, a system for GUI automation that makes task plans visible and editable by users, allowing them to supervise, correct, or intervene in the system's behavior. The system supports natural language guidance and screenshot-grounded interventions while preserving prior progress during repair.

What the Document Says

  • The paper discusses Plover, a vision-based GUI automation system.
  • Plover aims to address challenges in real-world GUI automation where dynamic layouts, unexpected dialogs, and evolving interface states can cause agents to drift from user intent.
  • Plover externalizes task plans and replanning as persistent, inspectable, and revisable artifacts.
  • It has a planner--executor architecture.
  • The system supports explicit supervision of evolving execution, localized correction through editable plans, natural-language guidance, and screenshot-grounded interventions.
  • Plover preserves prior progress during repair.
  • A formative study with six participants informed the interaction design.
  • Evaluation was done through benchmark failure-case repair and scenario-based workflow analyses.
  • Results show that many GUI automation failures are structurally repairable when plans remain visible and interventions are localized.
  • Explicit replanning helps make GUI automation more transparent, controllable, and adaptable.

Key Concepts Mentioned

  • GUI automation
  • plan-centric interaction
  • task plans
  • replanning
  • dynamic layouts
  • unexpected dialogs
  • evolving interface states

Claims Made by the Document

  • Plover addresses challenges in real-world GUI automation.
  • Externalizing task plans and replanning improves flexibility in GUI automation.
  • Plover supports explicit supervision, localized correction, natural-language guidance, and screenshot-grounded interventions.
  • Plover preserves prior progress during repair.
  • Many GUI automation failures are structurally repairable when plans remain visible and interventions are localized.
  • Explicit replanning helps make GUI automation more transparent, controllable, and adaptable.

Evidence Used

  • The paper describes Plover's design and features.
  • Results from a formative study and evaluations are presented.

Methods / Approach

  • Formative study with six participants
  • Benchmark failure-case repair
  • Scenario-based workflow analyses

Limitations Stated by the Authors

Not stated in source

Limitations Noted by the Agent

Agent interpretation — not from the source.

The paper does not provide information on the performance of Plover compared to other GUI automation systems.

Open Questions

  • What is the efficiency and effectiveness of Plover compared to existing GUI automation systems?
  • How scalable is Plover in handling complex and large-scale GUI tasks?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper focuses on the design and evaluation of Plover, but it does not provide a direct comparison with other GUI automation systems.

Raw Abstract / Excerpt

Graphical user interface (GUI) automation remains challenging in real-world environments, where dynamic layouts, unexpected dialogs, and evolving interface states can cause autonomous agents to drift from user intent. Recent vision-based multimodal agents improve flexibility by operating directly over screenshots and natural language instructions, but planning and adaptation often remain internal, limiting users' ability to inspect, supervise, or correct system behavior. We present Plover, a plan-centric vision-based GUI automation system that externalizes task plans and replanning as persistent, inspectable, and revisable artifacts. Through a planner--executor architecture, Plover supports explicit supervision of evolving execution, localized correction through editable plans, natural-language guidance, and screenshot-grounded interventions, while preserving prior progress during repair. A formative study with six participants informed the interaction design. We then evaluate Plover through benchmark failure-case repair and scenario-based workflow analyses. Our results show that many autonomous GUI-agent failures are structurally repairable when plans remain visible and interventions are localized, and that explicit replanning helps make GUI automation more transparent, controllable, and adaptable.