Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization

Source

arXiv

arxiv.org/abs/2607.15205v1

Read the source

Source

Plain-Language Brief

The paper introduces MM-IssueLoc, a new benchmark and evaluation protocol for repository-level issue localization with visual evidence. The benchmark contains over 650 issue-PR instances across multiple languages, annotated with various image categories and relevance levels. It also provides gold labels at file and function levels, and offers textual evidence conversion for images. Results show that existing systems struggle with multimodal repository localization, but MM-IssueLoc could help future research improve by explicitly evaluating the use of visual evidence.

What the Document Says

  • The paper presents MM-IssueLoc, a controlled benchmark and evaluation protocol for issue localization in repositories that include visual evidence such as screenshots, error dialogs, rendered UI states, and logs.
  • MM-IssueLoc consists of 652 issue-PR instances across 23 languages with annotations for 7 image categories and 4 relevance levels.
  • It provides gold labels at both file and function levels, offers paired text-only and with-image evaluations, and has VCE-based diagnostics that convert images into structured textual evidence.
  • The paper evaluates several systems, including LLM-based and retrieval-based systems, such as MM-IssueLoc-VL-Emb.
  • Results indicate that existing systems are not yet reliable for multimodal repository localization. The strongest agent achieves 38.96 file Acc@5 and 22.45 function Acc@10, while the best retriever reaches 33.86 function Acc@10.
  • Comparisons between benchmarks show that high localization scores on text-dominant SWE benchmarks do not easily transfer to multimodal issue localization.
  • MM-IssueLoc aims to make visual evidence an explicit evaluation variable for future research, allowing testing of whether systems can improve by using visual evidence for localization rather than relying solely on text-only cues or downstream patch-generation effects.

Key Concepts Mentioned

  • MM-IssueLoc
  • repository-level issue localization
  • visual evidence
  • image categories
  • relevance levels
  • file labels
  • function labels
  • text-only evaluation
  • with-image evaluation
  • VCE-based diagnostics

Claims Made by the Document

  • Existing systems are not reliable for multimodal repository localization.
  • High localization scores on text-dominant SWE benchmarks do not easily transfer to multimodal issue localization.

Evidence Used

  • The results of evaluating several systems, including LLM-based and retrieval-based systems, indicate that existing systems are not yet reliable for multimodal repository localization.
  • Cross-benchmark comparisons between MM-IssueLoc and text-dominant SWE benchmarks show that high localization scores do not easily transfer.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Real repository issues routinely include visual evidence such as screenshots, error dialogs, rendered UI states, and logs, yet repository-level issue localization is evaluated mostly as a text-only task. Existing multimodal SE benchmarks evaluate end-to-end repair, entangling localization with patch synthesis and obscuring whether visual input helped, hurt, or was ignored. We introduce \textbf{MM-IssueLoc}, a controlled benchmark and evaluation protocol for repository-level localization with visual evidence. MM-IssueLoc contains 652 issue-PR instances across 23 languages, with annotations for 7 image categories and 4 relevance levels. It provides file-level and function-level gold labels, paired text-only and with-image evaluation, and VCE-based diagnostics that convert images into structured textual evidence. We evaluate LLM-based and retrieval-based systems, including MM-IssueLoc-VL-Emb as a controlled multimodal retriever. Results show that existing systems remain far from reliable multimodal repository localization: the strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10. Cross-benchmark comparisons show that high localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization. MM-IssueLoc turns visual evidence into an explicit evaluation variable, enabling future work to test whether systems improve by using visual evidence for localization, rather than by relying on text-only cues or downstream patch-generation effects.