SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Zhou Liu, Ligang Huang, Zeli Su, Zewei Pan, Zhaoyang Han, Xing Chen, Yuanfeng Song, Wentao Zhang
• Venue: arXiv: cs.AI
• Published: 2026-08-11
• Collected: 2026-08-12
• URL: http://arxiv.org/abs/2608.10775v1
• PDF: https://arxiv.org/pdf/2608.10775v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

Researchers introduce Visual Skill Cards (VSCs), a new method for improving the decision-making abilities of computer agents in GUI environments. VSCs are memory representations that bind reusable procedures with visual evidence, applicability cues, and verification signals, helping agents understand which controls to use, when, and why. SkillLens constructs these cards from interaction experience and selectively expands the needed evidence at inference time for grounded GUI action prediction. Additionally, SkillLens supports CardDistill, a method that uses VSC evidence as privileged context to train student models that can act without runtime card retrieval.

What the Document Says

• The authors introduce Visual Skill Cards (VSCs) for computer agents in GUI environments.
• VSCs bind reusable procedures with visual evidence, applicability cues, and verification signals.
• SkillLens constructs VSCs from interaction experience and retrieves relevant cards for grounded GUI action prediction.
• SkillLens supports CardDistill, which uses VSC evidence as privileged context for training student models.
• Improvements in Step SR and Overall metrics were observed on Multimodal-Mind2Web and WebLINX-BrowserGym datasets.

Key Concepts Mentioned

• Visual Skill Cards (VSCs)
• SkillLens
• GUI action prediction
• CardDistill

Claims Made by the Document

• VSCs improve the decision-making abilities of computer agents in GUI environments.
• SkillLens improves the performance of a GPT-5.4-mini executor on Multimodal-Mind2Web and WebLINX-BrowserGym datasets.

Evidence Used

• Improvements in Step SR and Overall metrics were observed...

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Computer-using agents can perceive rich software interfaces, yet their decisions often lack visual procedural memory: they may recognize individual controls without identifying which familiar workflow is active, which control matters next, or what evidence would confirm progress. Raw interaction traces preserve such information but are long and noisy to condition on, whereas text-only skills often omit the visual state that makes a procedure applicable. We introduce Visual Skill Cards (VSCs), a state-conditioned memory representation that binds reusable procedures with applicability cues, visual evidence, and verification signals. SkillLens constructs VSCs from heterogeneous interaction experience through Trace-to-Visual-Skill-Card and, at inference time, retrieves relevant cards and selectively expands only the evidence needed by a fixed visual-language model executor for grounded GUI action prediction. The same representation also supports CardDistill, which uses VSC evidence as privileged teacher context to train a student that acts without runtime card retrieval. Across Multimodal-Mind2Web and WebLINX-BrowserGym, SkillLens improves the frozen GPT-5.4-mini executor by +11.6 points in Step SR and +2.9 points in Overall, respectively; CardDistill further improves the corresponding student-only Qwen3-VL-2B metrics by +12.0 and +3.2 points.