UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Grace Man Chen, Litao Guo, Yifan Wu, Yiyu Chen, Yenchi Tseng, Sicheng Liu, Yuyu Luo, Ying-Cong Chen
• Venue: arXiv: cs.SE
• Published: 2026-07-07
• Collected: 2026-07-08
• URL: http://arxiv.org/abs/2607.06306v1
• PDF: https://arxiv.org/pdf/2607.06306v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers have introduced UI2App, a new benchmark focused on evaluating the ability of large language models to recover application behavior from screenshots alone, without any textual or behavioral guidance. The benchmark includes 327 screenshots grouped into 45 sets for runnable multi-route web applications and assesses each artifact along four dimensions: executability, navigation reachability, visual fidelity, and interaction inference. The results show that while models can reconstruct visuals well, they struggle to infer complex interactions, such as cross-page state management.

What the Document Says

• UI2App is a benchmark targeting interaction inference.
• It consists of 327 screenshots organized into 45 sets for runnable multi-route web applications.
• The benchmark evaluates each artifact along four dimensions: executability, navigation reachability, visual fidelity, and interaction inference.
• The interaction metric (IIS) assesses inferred interactions by functional correctness and state-management complexity.
• Experiments on six frontier vision-language models reveal a mismatch between visual reconstruction and interaction realization.
• The leader in visual fidelity scores only 7.5 on IIS, ranking fourth and trailing the IIS leader by 5.2x.
• High-complexity interactions such as cross-page state remain a pervasive bottleneck, with half of the evaluated models scoring zero on this dimension.

Key Concepts Mentioned

• UI2App
• benchmark
• interaction inference
• screenshots
• web applications
• large language models
• vision-language models
• visual fidelity
• IIS

Claims Made by the Document

• UI2App is the first benchmark targeting interaction inference.
• Half of the evaluated models struggle with high-complexity interactions such as cross-page state management.

Evidence Used

• UI2App comprises 327 screenshots grouped into 45 state-coherent screenshot sets for runnable multi-route web applications
• Experiments on six frontier vision-language models

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex prompts that impose substantial demands on users and offer limited expressivity for page layout and cross-page visual coherence. Image-driven paradigms, which take UI screenshots as input, align more closely with real development workflows. However, current benchmarks focus primarily on visual fidelity and lack a systematic evaluation of the interaction capabilities in generated artifacts. To address this gap, we introduce UI2App, the first benchmark targeting interaction inference, the ability to recover application behavior from screenshots alone, without any textual or behavioral guidance. UI2App comprises 327 screenshots grouped into 45 state-coherent screenshot sets for runnable multi-route web applications. We design an end-to-end pipeline that evaluates each artifact along four dimensions: executability, navigation reachability, visual fidelity, and interaction inference. The interaction metric (IIS) assesses inferred interactions by functional correctness and state-management complexity, crediting any valid implementation rather than matching a single reference. Experiments on six frontier vision-language models reveal a marked capability mismatch between visual reconstruction and interaction realization: the visual-fidelity leader scores only 7.5 on IIS, ranking fourth and trailing the IIS leader by 5.2x. High-complexity interactions such as cross-page state remain a pervasive bottleneck, with half of the evaluated models scoring exactly zero on this dimension. Overall, the results indicate that inferring complete interaction behavior from static screenshots remains a key challenge for models.