MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Shaoxiong Zhan, Shi Hu, Boyu Feng, Hai Lin, Andrew Gong, Zhengda Zhou, Jiaying Zhou, Yunyun Hou, Hao Su, Hai-Tao Zheng
• Venue: arXiv: cs.SE
• Published: 2026-07-16
• Collected: 2026-07-17
• URL: http://arxiv.org/abs/2607.15205v1
• PDF: https://arxiv.org/pdf/2607.15205v1
• DOI: UNKNOWN

Plain-Language Brief

The paper introduces MM-IssueLoc, a benchmark and evaluation protocol for repository-level issue localization that includes visual evidence. The benchmark has 652 issue-PR instances across 23 languages with annotations for 7 image categories and 4 relevance levels. It provides file-level and function-level gold labels, paired text-only and with-image evaluation, and VCE-based diagnostics that convert images into structured textual evidence. The results show that existing systems struggle with multimodal repository localization, with the strongest agent reaching 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10.

What the Document Says

• The paper presents MM-IssueLoc, a benchmark and evaluation protocol for repository-level issue localization with visual evidence.
• MM-IssueLoc contains 652 issue-PR instances across 23 languages.
• It provides annotations for 7 image categories and 4 relevance levels.
• The benchmark offers file-level and function-level gold labels.
• It provides paired text-only and with-image evaluation.
• MM-IssueLoc uses VCE-based diagnostics to convert images into structured textual evidence.
• The paper evaluates LLM-based and retrieval-based systems, including MM-IssueLoc-VL-Emb as a controlled multimodal retriever.
• Results show that existing systems struggle with multimodal repository localization.
• The strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10.
• The strongest retriever reaches 33.86 function Acc@10.

Key Concepts Mentioned

• MM-IssueLoc
• repository-level issue localization
• visual evidence
• benchmark
• evaluation protocol
• LLM
• retrieval-based systems

Claims Made by the Document

• Existing multimodal SE benchmarks evaluate end-to-end repair, obscuring whether visual input helped, hurt, or was ignored.
• The strongest agent and retriever struggle with multimodal repository localization.
• High localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization.

Evidence Used

• MM-IssueLoc introduces a controlled benchmark and evaluation protocol for repository-level localization with visual evidence.
• Results show the performance of existing systems in multimodal repository localization.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Real repository issues routinely include visual evidence such as screenshots, error dialogs, rendered UI states, and logs, yet repository-level issue localization is evaluated mostly as a text-only task. Existing multimodal SE benchmarks evaluate end-to-end repair, entangling localization with patch synthesis and obscuring whether visual input helped, hurt, or was ignored. We introduce \textbf{MM-IssueLoc}, a controlled benchmark and evaluation protocol for repository-level localization with visual evidence. MM-IssueLoc contains 652 issue-PR instances across 23 languages, with annotations for 7 image categories and 4 relevance levels. It provides file-level and function-level gold labels, paired text-only and with-image evaluation, and VCE-based diagnostics that convert images into structured textual evidence. We evaluate LLM-based and retrieval-based systems, including MM-IssueLoc-VL-Emb as a controlled multimodal retriever. Results show that existing systems remain far from reliable multimodal repository localization: the strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10. Cross-benchmark comparisons show that high localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization. MM-IssueLoc turns visual evidence into an explicit evaluation variable, enabling future work to test whether systems improve by using visual evidence for localization, rather than by relying on text-only cues or downstream patch-generation effects.