SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Zhou Liu, Ligang Huang, Zeli Su, Zewei Pan, Zhaoyang Han, Xing Chen, Yuanfeng Song, Wentao Zhang
• Venue: arXiv: cs.AI
• Published: 2026-08-11
• Collected: 2026-08-12
• URL: http://arxiv.org/abs/2608.10775v1
• PDF: https://arxiv.org/pdf/2608.10775v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

The paper introduces Visual Skill Cards (VSCs), a memory representation that helps computer agents make decisions by binding reusable procedures with applicability cues, visual evidence, and verification signals. The system, called SkillLens, constructs VSCs from past interaction experience and uses them to improve the performance of a GPT-5.4-mini executor in multitasking and overall tasks on two datasets.

What the Document Says

• The paper presents Visual Skill Cards (VSCs) as a state-conditioned memory representation for computer agents.
• VSCs bind reusable procedures with applicability cues, visual evidence, and verification signals.
• SkillLens constructs VSCs from heterogeneous interaction experience.
• At inference time, SkillLens retrieves relevant cards and uses them to guide a fixed visual-language model executor for GUI action prediction.
• SkillLens also supports CardDistill, which uses VSC evidence to train a student that can act without runtime card retrieval.
• The paper reports improvements in the frozen GPT-5.4-mini executor's performance on two datasets: Multimodal-Mind2Web and WebLINX-BrowserGym.

Key Concepts Mentioned

• Visual Skill Cards (VSCs)
• SkillLens
• Trace-to-Visual-Skill-Card
• GUIs
• computer agents
• retrieval-augmented GUI action prediction

Claims Made by the Document

• VSCs improve the performance of a GPT-5.4-mini executor.
• CardDistill improves the performance of a student-only Qwen3-VL-2B.

Evidence Used

• +11.6 points in Step SR and +2.9 points in Overall on Multimodal-Mind2Web
• +12.0 and +3.2 points on corresponding student-only Qwen3-VL-2B metrics.

Methods / Approach

• Trace-to-Visual-Skill-Card
• retrieval-augmented GUI action prediction
• CardDistill

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Computer-using agents can perceive rich software interfaces, yet their decisions often lack visual procedural memory: they may recognize individual controls without identifying which familiar workflow is active, which control matters next, or what evidence would confirm progress. Raw interaction traces preserve such information but are long and noisy to condition on, whereas text-only skills often omit the visual state that makes a procedure applicable. We introduce Visual Skill Cards (VSCs), a state-conditioned memory representation that binds reusable procedures with applicability cues, visual evidence, and verification signals. SkillLens constructs VSCs from heterogeneous interaction experience through Trace-to-Visual-Skill-Card and, at inference time, retrieves relevant cards and selectively expands only the evidence needed by a fixed visual-language model executor for grounded GUI action prediction. The same representation also supports CardDistill, which uses VSC evidence as privileged teacher context to train a student that acts without runtime card retrieval. Across Multimodal-Mind2Web and WebLINX-BrowserGym, SkillLens improves the frozen GPT-5.4-mini executor by +11.6 points in Step SR and +2.9 points in Overall, respectively; CardDistill further improves the corresponding student-only Qwen3-VL-2B metrics by +12.0 and +3.2 points.