MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Shaoxiong Zhan, Shi Hu, Boyu Feng, Hai Lin, Andrew Gong, Zhengda Zhou, Jiaying Zhou, Yunyun Hou, Hao Su, Hai-Tao Zheng
• Venue: arXiv: cs.SE
• Published: 2026-07-16
• Collected: 2026-07-17
• URL: http://arxiv.org/abs/2607.15205v1
• PDF: https://arxiv.org/pdf/2607.15205v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers introduce MM-IssueLoc, a benchmark and evaluation protocol designed specifically for repository-level issue localization with visual evidence. The dataset includes 652 issue-PR instances across multiple languages, annotated with various image categories and relevance levels. It provides file-level and function-level gold labels, as well as textual evidence derived from images using VCE. Results show that existing systems struggle with reliable multimodal repository localization, but MM-IssueLoc may help in testing whether future systems improve by utilizing visual evidence for localization rather than relying on text-only cues.

What the Document Says

• MM-IssueLoc is a new benchmark and evaluation protocol for multimodal repository-level issue localization.
• The dataset consists of 652 issue-PR instances across 23 languages with annotations for 7 image categories and 4 relevance levels.
• It provides file-level and function-level gold labels, as well as textual evidence derived from images using VCE.
• Results show that existing systems struggle with reliable multimodal repository localization.
• The strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10.

Key Concepts Mentioned

• repository-level issue localization
• multimodal SE benchmarks
• MM-IssueLoc
• visual evidence
• VCE
• LLM
• retrieval-based systems

Claims Made by the Document

• Existing multimodal SE benchmarks do not provide a clear evaluation of visual input.
• MM-IssueLoc enables testing whether future systems improve by using visual evidence for localization.

Evidence Used

• The text mentions that existing multimodal SE benchmarks evaluate end-to-end repair, entangling localization with patch synthesis and obscuring whether visual input helped or was ignored.
• Results from MM-IssueLoc show the limitations of existing systems in reliable multimodal repository localization.

Methods / Approach

• The text does not explicitly mention the methods used in the research.

Limitations Stated by the Authors

The authors acknowledge that their results show the limitations of existing systems but do not provide a solution or suggest improvements.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The agent's limitation is that it cannot perform experiments or suggest solutions based on the provided text.

Open Questions

• What are the specific image categories and relevance levels in MM-IssueLoc?
• How does the VCE method work to convert images into structured textual evidence?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The text provides some information about the dataset, evaluation methods, and results but lacks details on specific aspects such as image categories and VCE.

Raw Abstract / Excerpt

Real repository issues routinely include visual evidence such as screenshots, error dialogs, rendered UI states, and logs, yet repository-level issue localization is evaluated mostly as a text-only task. Existing multimodal SE benchmarks evaluate end-to-end repair, entangling localization with patch synthesis and obscuring whether visual input helped, hurt, or was ignored. We introduce \textbf{MM-IssueLoc}, a controlled benchmark and evaluation protocol for repository-level localization with visual evidence. MM-IssueLoc contains 652 issue-PR instances across 23 languages, with annotations for 7 image categories and 4 relevance levels. It provides file-level and function-level gold labels, paired text-only and with-image evaluation, and VCE-based diagnostics that convert images into structured textual evidence. We evaluate LLM-based and retrieval-based systems, including MM-IssueLoc-VL-Emb as a controlled multimodal retriever. Results show that existing systems remain far from reliable multimodal repository localization: the strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10. Cross-benchmark comparisons show that high localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization. MM-IssueLoc turns visual evidence into an explicit evaluation variable, enabling future work to test whether systems improve by using visual evidence for localization, rather than by relying on text-only cues or downstream patch-generation effects.