Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

WinDOM: Self-Family Distillation for Small-Model GUI Grounding

Source

arXiv

arxiv.org/abs/2606.25964v1

Read the source

Source

Plain-Language Brief

Researchers present WinDOM, a large dataset of bounding box training data for small GUI-grounding agents, which can be used for on-device deployment, accessibility tooling, and low-cost iteration. They also introduce Self-Family Distillation (SFD), a method that addresses the challenge of obtaining bounding box data without expensive human annotation and combines supervised fine-tuning with reinforcement learning. SFD is shown to improve the performance of small models on various tasks.

What the Document Says

  • WinDOM is a dataset of grounding corpus harvested from Windows 11 web reimplementation under headless Playwright, containing 54,425 records and no OCR or human annotation.
  • Small GUI-grounding agents face two open recipe questions: obtaining bounding box training data without expensive human annotation and combining supervised fine-tuning with reinforcement learning.
  • Self-Family Distillation (SFD) is a method that addresses these questions, requiring only the teacher choice as a parameter.
  • SFD can be initiated either using an EMA of the student (no external model) or a frozen larger same-family teacher.
  • The saturation depth of the SFD cold-start is treated as an explicit GRPO hyperparameter.
  • On a Qwen3.5-2B student, the under-saturated cold-start is shown to be a better GRPO initializer than the converged one.
  • SFD-4B with Early-init RL improves performance over the base by +5.4 OOD-mean and +3.5 on ScreenSpot-Pro, +7.0 on OSWorld-G, and +5.8 on ScreenSpot-V2.
  • The same-size EMA mode of SFD lands within roughly one OOD-mean point of the cross-size 4B variant.

Key Concepts Mentioned

  • WinDOM
  • GUI-grounding agents
  • bounding box training data
  • Small models
  • Self-Family Distillation (SFD)
  • GRPO hyperparameter
  • OOD-mean
  • ScreenSpot-Pro
  • OSWorld-G
  • ScreenSpot-V2

Claims Made by the Document

  • WinDOM is a large dataset of grounding corpus.
  • Small GUI-grounding agents face two open recipe questions.
  • SFD addresses these questions and improves the performance of small models.
  • Under-saturated SFD cold-start is a better GRPO initializer than the converged one on Qwen3.5-2B student.
  • SFD-4B with Early-init RL improves performance over the base.

Evidence Used

  • WinDOM is harvested from Windows 11 web reimplementation under headless Playwright, containing 54,425 records.
  • The text mentions the two open recipe questions faced by small GUI-grounding agents and SFD as a solution.
  • The results demonstrate that under-saturated SFD cold-start outperforms the converged one on Qwen3.5-2B student.
  • The improvements in performance of SFD-4B with Early-init RL are stated.

Methods / Approach

  • WinDOM dataset harvesting
  • Self-Family Distillation (SFD)

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Small ($\sim$2B) GUI-grounding agents are attractive for on-device deployment, accessibility tooling, and low-cost iteration, but at this scale they face two open recipe questions: how to obtain bounding-box training data without expensive human annotation, and how to combine supervised fine-tuning with reinforcement learning. We address both, with the explicit goal of pushing small-model performance rather than scaling up. WinDOM is a $54{,}425$-record grounding corpus harvested by driving an open-source Windows 11 web reimplementation under headless Playwright, with bounding boxes read directly off the DOM and no OCR or human annotation. Self-Family Distillation (SFD) is a single rejection-sampling cold-start parameterised only by the teacher choice: either an EMA of the student (no external model) or a frozen larger same-family teacher. We then treat the saturation depth of the SFD cold-start as an explicit GRPO hyperparameter. On a Qwen3.5-2B student, the under-saturated cold-start is a better GRPO initialiser than the converged one: SFD-4B with Early-init RL gains $+5.4$ OOD-mean ($+3.5$ ScreenSpot-Pro, $+7.0$ OSWorld-G, $+5.8$ ScreenSpot-V2) over the base. The same-size EMA mode lands within roughly one OOD-mean point of the cross-size $4$B variant ($65.2$ vs $66.3$) without an external teacher.