MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization

Source

Source name: arXiv
Source kind: paper
Authors / organization: Shaoxiong Zhan, Shi Hu, Boyu Feng, Hai Lin, Andrew Gong, Zhengda Zhou, Jiaying Zhou, Yunyun Hou, Hao Su, Hai-Tao Zheng
Venue: arXiv: cs.SE
Published: 2026-07-16
Collected: 2026-07-17
URL: http://arxiv.org/abs/2607.15205v1
PDF: https://arxiv.org/pdf/2607.15205v1
DOI: UNKNOWN

Plain-Language Brief

Researchers have introduced a new benchmark called MM-IssueLoc for evaluating the effectiveness of systems in repository-level issue localization that also includes visual evidence such as screenshots, error dialogs, and logs. The benchmark contains 652 issues in 23 languages, with annotations for 7 image categories and 4 relevance levels. It provides gold labels at file level and function level, allows evaluation of systems both with and without images, and offers diagnostics that convert images into structured textual evidence. Results show that existing systems are not yet reliable for multimodal repository localization.

What the Document Says

MM-IssueLoc is a new benchmark for evaluating repository-level issue localization with visual evidence.
It contains 652 issue-PR instances across 23 languages, annotated for 7 image categories and 4 relevance levels.
It provides file-level and function-level gold labels.
Evaluation can be done both with and without images.
It offers VCE-based diagnostics to convert images into structured textual evidence.
Results show that existing systems are not reliable for multimodal repository localization.

Key Concepts Mentioned

Repository-level issue localization
Visual evidence
MM-IssueLoc benchmark
Image categories
Relevance levels
Gold labels
Evaluation
VCE-based diagnostics

Claims Made by the Document

Existing systems are not reliable for multimodal repository localization.
High localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization.

Evidence Used

Results show that the strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10.
Cross-benchmark comparisons show that high localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Real repository issues routinely include visual evidence such as screenshots, error dialogs, rendered UI states, and logs, yet repository-level issue localization is evaluated mostly as a text-only task. Existing multimodal SE benchmarks evaluate end-to-end repair, entangling localization with patch synthesis and obscuring whether visual input helped, hurt, or was ignored. We introduce \textbf{MM-IssueLoc}, a controlled benchmark and evaluation protocol for repository-level localization with visual evidence. MM-IssueLoc contains 652 issue-PR instances across 23 languages, with annotations for 7 image categories and 4 relevance levels. It provides file-level and function-level gold labels, paired text-only and with-image evaluation, and VCE-based diagnostics that convert images into structured textual evidence. We evaluate LLM-based and retrieval-based systems, including MM-IssueLoc-VL-Emb as a controlled multimodal retriever. Results show that existing systems remain far from reliable multimodal repository localization: the strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10. Cross-benchmark comparisons show that high localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization. MM-IssueLoc turns visual evidence into an explicit evaluation variable, enabling future work to test whether systems improve by using visual evidence for localization, rather than by relying on text-only cues or downstream patch-generation effects.