MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Shaoxiong Zhan, Shi Hu, Boyu Feng, Hai Lin, Andrew Gong, Zhengda Zhou, Jiaying Zhou, Yunyun Hou, Hao Su, Hai-Tao Zheng
• Venue: arXiv: cs.SE
• Published: 2026-07-16
• Collected: 2026-07-17
• URL: http://arxiv.org/abs/2607.15205v1
• PDF: https://arxiv.org/pdf/2607.15205v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers have introduced a new benchmark called MM-IssueLoc for evaluating the effectiveness of multimodal repository-level issue localization, which includes visual evidence such as screenshots, error dialogs, and logs. The benchmark contains 652 issue-PR instances across 23 languages with annotations for various image categories and relevance levels. The researchers evaluate different systems, including a controlled multimodal retriever, and find that existing systems struggle with reliable multimodal repository localization.

What the Document Says

• Researchers have introduced MM-IssueLoc, a benchmark for evaluating the effectiveness of multimodal repository-level issue localization
• MM-IssueLoc contains 652 issue-PR instances across 23 languages with annotations for various image categories and relevance levels
• The benchmark provides file-level and function-level gold labels, paired text-only and with-image evaluation, and VCE-based diagnostics that convert images into structured textual evidence
• The researchers evaluate LLM-based and retrieval-based systems on MM-IssueLoc
• The strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10
• Existing systems struggle with reliable multimodal repository localization

Key Concepts Mentioned

• repository-level issue localization
• multimodal SE benchmarks
• visual evidence
• MM-IssueLoc
• LLM
• retrieval-based systems

Claims Made by the Document

• Existing multimodal SE benchmarks evaluate end-to-end repair, entangling localization with patch synthesis and obscuring whether visual input helped or was ignored
• MM-IssueLoc enables future work to test whether systems improve by using visual evidence for localization, rather than relying on text-only cues or downstream patch-generation effects

Evidence Used

• Existing multimodal SE benchmarks are evaluated as a text-only task
• MM-IssueLoc contains annotations for various image categories and relevance levels
• Results from the evaluation of LLM-based and retrieval-based systems on MM-IssueLoc

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Real repository issues routinely include visual evidence such as screenshots, error dialogs, rendered UI states, and logs, yet repository-level issue localization is evaluated mostly as a text-only task. Existing multimodal SE benchmarks evaluate end-to-end repair, entangling localization with patch synthesis and obscuring whether visual input helped, hurt, or was ignored. We introduce \textbf{MM-IssueLoc}, a controlled benchmark and evaluation protocol for repository-level localization with visual evidence. MM-IssueLoc contains 652 issue-PR instances across 23 languages, with annotations for 7 image categories and 4 relevance levels. It provides file-level and function-level gold labels, paired text-only and with-image evaluation, and VCE-based diagnostics that convert images into structured textual evidence. We evaluate LLM-based and retrieval-based systems, including MM-IssueLoc-VL-Emb as a controlled multimodal retriever. Results show that existing systems remain far from reliable multimodal repository localization: the strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10. Cross-benchmark comparisons show that high localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization. MM-IssueLoc turns visual evidence into an explicit evaluation variable, enabling future work to test whether systems improve by using visual evidence for localization, rather than by relying on text-only cues or downstream patch-generation effects.