Plover: Steering GUI Agents through Plan-Centric Interaction

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Madhumitha Venkatesan, Shicheng Wen, Jiajing Guo, Jorge Piazentin Ono, Liu Ren, Dongyu Liu
• Venue: arXiv: cs.AI
• Published: 2026-07-16
• Collected: 2026-07-17
• URL: http://arxiv.org/abs/2607.15193v1
• PDF: https://arxiv.org/pdf/2607.15193v1
• DOI: UNKNOWN

Plain-Language Brief

The paper introduces Plover, a system for GUI automation in real-world environments. It focuses on improving flexibility and user control by making task plans persistent, inspectable, and editable, while also allowing natural-language guidance and screenshot-grounded interventions.

What the Document Says

• The paper discusses a system called Plover for automating Graphical User Interfaces (GUIs).
• Challenges in real-world environments include dynamic layouts, unexpected dialogs, and evolving interface states that can cause autonomous agents to deviate from user intent.
• Vision-based multimodal agents offer improved flexibility by operating on screenshots and natural language instructions, but planning and adaptation are typically internal.
• Plover aims to address these issues by externalizing task plans and replanning as persistent, inspectable, and revisable artifacts.
• The system uses a planner-executor architecture for explicit supervision of evolving execution, localized correction through editable plans, natural-language guidance, and screenshot-grounded interventions.
• Prior progress is preserved during repair in Plover.
• The authors conducted a formative study with six participants to inform the interaction design.
• Evaluation was done through benchmark failure-case repair and scenario-based workflow analyses.
• Results suggest that many GUI automation failures can be structurally repaired when plans remain visible and interventions are localized, and explicit replanning helps make GUI automation more transparent, controllable, and adaptable.

Key Concepts Mentioned

• GUI automation
• dynamic layouts
• unexpected dialogs
• evolving interface states
• vision-based multimodal agents
• planner-executor architecture
• task plans
• replanning
• natural-language guidance
• screenshot-grounded interventions

Claims Made by the Document

• Plover improves flexibility in real-world GUI automation.
• Externalizing task plans and replanning helps make GUI automation more transparent, controllable, and adaptable.

Evidence Used

• Participants were informed for interaction design
• Evaluation was done through benchmark failure-case repair and scenario-based workflow analyses
• Results show that explicit replanning helps make GUI automation more transparent, controllable, and adaptable

Methods / Approach

• Formative study with six participants
• Benchmark failure-case repair
• Scenario-based workflow analyses

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

The data does not provide information about potential limitations as stated by the authors.

Open Questions

• What specific approaches does Plover use for externalizing task plans and replanning?
• How does Plover handle complex or unanticipated situations in real-world environments?

Notes for Future Q&A

May include interpretation. Labeled where it does.

It would be interesting to learn more about the specific techniques used by Plover for externalizing task plans, and its performance in handling unexpected scenarios in real-world GUI automation.

Raw Abstract / Excerpt

Graphical user interface (GUI) automation remains challenging in real-world environments, where dynamic layouts, unexpected dialogs, and evolving interface states can cause autonomous agents to drift from user intent. Recent vision-based multimodal agents improve flexibility by operating directly over screenshots and natural language instructions, but planning and adaptation often remain internal, limiting users' ability to inspect, supervise, or correct system behavior. We present Plover, a plan-centric vision-based GUI automation system that externalizes task plans and replanning as persistent, inspectable, and revisable artifacts. Through a planner--executor architecture, Plover supports explicit supervision of evolving execution, localized correction through editable plans, natural-language guidance, and screenshot-grounded interventions, while preserving prior progress during repair. A formative study with six participants informed the interaction design. We then evaluate Plover through benchmark failure-case repair and scenario-based workflow analyses. Our results show that many autonomous GUI-agent failures are structurally repairable when plans remain visible and interventions are localized, and that explicit replanning helps make GUI automation more transparent, controllable, and adaptable.