Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution

Source

arXiv

arxiv.org/abs/2607.00700v1

Read the source

Source

Plain-Language Brief

The authors present LLVM-Bench, a large-scale benchmark for testing the effectiveness of large language models (LLMs) in resolving issues related to the LLVM compiler. They found that current LLM-based techniques are limited on LLVM-Bench, with high rates of patch invalidity and build failures. To address this, they propose LLVM-Ens, a lightweight ensemble approach that improves the resolution rate by integrating patches from multiple techniques.

What the Document Says

  • The text discusses the use of large language models (LLMs) for issue resolution in the LLVM compiler infrastructure.
  • The authors introduce LLVM-Bench, a benchmark containing 423 real-world, validated tasks collected from the LLVM project.
  • They also present LLVM-Gym, a platform that automates various processes for evaluating LLMs on LLVM issues.
  • Four representative LLMs, six retrieval configurations, and three agents were studied using LLVM-Bench and LLVM-Gym.
  • The study revealed that current LLM-based techniques have limitations on LLVM-Bench, with high rates of patch invalidity and build failures.
  • A strong complementarity among different LLMs and agents was found.
  • The authors propose LLVM-Ens, an ensemble approach that improves the resolution rate by integrating patches from multiple techniques.

Key Concepts Mentioned

  • LLVM
  • large language models (LLMs)
  • issue resolution
  • LLVM-Bench
  • LLVM-Gym
  • patch invalidity
  • build failures
  • ensemble approach

Claims Made by the Document

  • Current LLM-based issue resolution techniques are limited on LLVM-Bench.
  • There is a strong complementarity among different LLMs and agents.

Evidence Used

  • results show that current LLM-based issue resolution techniques remain limited on LLVM-Bench
  • we further reveal a strong complementarity among different LLMs and agents

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging. Although large language models (LLMs) have recently achieved remarkable success in issue resolution, their effectiveness on complex system-level LLVM compiler remains largely unexplored. To address this gap, we introduce LLVM-Bench, the first large-scale benchmark for LLVM issue resolution, containing 423 real-world, validated tasks collected from the LLVM project. We further develop LLVM-Gym, a scalable evaluation platform that automates issue reproduction, patch application, compiler building, and test execution. Using LLVM-Bench and LLVM-Gym, we conduct a comprehensive study of four representative LLMs, six retrieval configurations, and three agents. Our results show that current LLM-based issue resolution techniques remain limited on LLVM-Bench, with patch invalidity and build failures as the dominant failure modes. We further reveal a strong complementarity among different LLMs and agents, motivating LLVM-Ens, a lightweight ensemble approach that expands the patch space through integrating the patches generated by diverse techniques, filters incorrect and redundant candidates, and identifies the most promising solution. Our results show that LLVM-Ens achieves a resolution rate of up to 21.99%, further improving LLVM issue resolution.