Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization

Source

arXiv

arxiv.org/abs/2607.15205v1

Read the source

Source

Plain-Language Brief

Researchers have created a new benchmark called MM-IssueLoc to evaluate the effectiveness of multimodal repository-level issue localization, which includes visual evidence such as screenshots and logs. Current evaluation methods primarily focus on text-only tasks, but this new benchmark provides annotations for images and allows for evaluations with and without image data. The study finds that existing systems are not yet reliable at multimodal repository localization, with the strongest agent achieving 38.96 file accuracy at 5 and 22.45 function accuracy at 10, and the strongest retriever reaching 33.86 function accuracy at 10. The study also suggests that high localization scores on text-dominant benchmarks do not always transfer well to multimodal issue localization.

What the Document Says

  • A new benchmark called MM-IssueLoc has been introduced for evaluating repository-level localization with visual evidence.
  • The benchmark contains 652 issue-PR instances across 23 languages, with annotations for 7 image categories and 4 relevance levels.
  • It provides file-level and function-level gold labels, paired text-only and with-image evaluation, and VCE-based diagnostics that convert images into structured textual evidence.
  • The study evaluates LLM-based and retrieval-based systems, including MM-IssueLoc-VL-Emb as a controlled multimodal retriever.
  • Results show that existing systems are not yet reliable at multimodal repository localization.
  • Cross-benchmark comparisons suggest that high localization scores on text-dominant benchmarks do not always transfer well to multimodal issue localization.

Key Concepts Mentioned

  • MM-IssueLoc
  • repository-level issue localization
  • visual evidence
  • benchmark
  • evaluation protocol
  • images
  • text-only
  • LLM-based systems
  • retrieval-based systems

Claims Made by the Document

  • Existing multimodal SE benchmarks evaluate end-to-end repair, obscuring whether visual input helped, hurt, or was ignored.
  • The strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10.

Evidence Used

  • The text mentions that existing multimodal SE benchmarks evaluate end-to-end repair and that the study introduces a new benchmark to address this issue.
  • The text presents results from evaluating LLM-based and retrieval-based systems on the new MM-IssueLoc benchmark.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Real repository issues routinely include visual evidence such as screenshots, error dialogs, rendered UI states, and logs, yet repository-level issue localization is evaluated mostly as a text-only task. Existing multimodal SE benchmarks evaluate end-to-end repair, entangling localization with patch synthesis and obscuring whether visual input helped, hurt, or was ignored. We introduce \textbf{MM-IssueLoc}, a controlled benchmark and evaluation protocol for repository-level localization with visual evidence. MM-IssueLoc contains 652 issue-PR instances across 23 languages, with annotations for 7 image categories and 4 relevance levels. It provides file-level and function-level gold labels, paired text-only and with-image evaluation, and VCE-based diagnostics that convert images into structured textual evidence. We evaluate LLM-based and retrieval-based systems, including MM-IssueLoc-VL-Emb as a controlled multimodal retriever. Results show that existing systems remain far from reliable multimodal repository localization: the strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10. Cross-benchmark comparisons show that high localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization. MM-IssueLoc turns visual evidence into an explicit evaluation variable, enabling future work to test whether systems improve by using visual evidence for localization, rather than by relying on text-only cues or downstream patch-generation effects.