Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

WinDOM: Self-Family Distillation for Small-Model GUI Grounding

Source

arXiv

arxiv.org/abs/2606.25964v1

Read the source

Source

Plain-Language Brief

Researchers have created a method called WinDOM for small GUI-grounding agents, which are useful for on-device deployment and accessibility tools. The method uses self-Family Distillation (SFD) to obtain bounding-box training data without expensive human annotation and to combine supervised fine-tuning with reinforcement learning. The study also presents a new grounding corpus of 54,425 records that do not require OCR or human annotation.

What the Document Says

  • Researchers have developed a method called WinDOM for small GUI-grounding agents.
  • The method uses Self-Family Distillation (SFD) to obtain bounding-box training data without OCR or human annotation and to combine supervised fine-tuning with reinforcement learning.
  • WinDOM is a grounding corpus of 54,425 records that were harvested by driving a Windows 11 web reimplementation under headless Playwright.
  • The study presents the results of using SFD on a Qwen3.5-2B student, showing improvements in various tasks over the base model.

Key Concepts Mentioned

  • WinDOM
  • Self-Family Distillation (SFD)
  • GUI-grounding agents
  • bounding-box training data
  • reinforcement learning
  • Windows 11 web reimplementation

Claims Made by the Document

  • WinDOM is an effective method for small GUI-grounding agents.
  • SFD can provide better initial performance than converged parameters for some tasks.

Evidence Used

  • The results of using SFD on a Qwen3.5-2B student are presented, showing improvements in various tasks over the base model.

Methods / Approach

  • WinDOM
  • Self-Family Distillation (SFD)
  • headless Playwright

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Small ($\sim$2B) GUI-grounding agents are attractive for on-device deployment, accessibility tooling, and low-cost iteration, but at this scale they face two open recipe questions: how to obtain bounding-box training data without expensive human annotation, and how to combine supervised fine-tuning with reinforcement learning. We address both, with the explicit goal of pushing small-model performance rather than scaling up. WinDOM is a $54{,}425$-record grounding corpus harvested by driving an open-source Windows 11 web reimplementation under headless Playwright, with bounding boxes read directly off the DOM and no OCR or human annotation. Self-Family Distillation (SFD) is a single rejection-sampling cold-start parameterised only by the teacher choice: either an EMA of the student (no external model) or a frozen larger same-family teacher. We then treat the saturation depth of the SFD cold-start as an explicit GRPO hyperparameter. On a Qwen3.5-2B student, the under-saturated cold-start is a better GRPO initialiser than the converged one: SFD-4B with Early-init RL gains $+5.4$ OOD-mean ($+3.5$ ScreenSpot-Pro, $+7.0$ OSWorld-G, $+5.8$ ScreenSpot-V2) over the base. The same-size EMA mode lands within roughly one OOD-mean point of the cross-size $4$B variant ($65.2$ vs $66.3$) without an external teacher.