LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution

Abstract: LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging. Although LLMs have recently achieved remarkable success in issue resolution, their effectiveness on complex system-level LLVM compiler remains largely unexplored. To address this gap, we introduce LLVM-Bench, the first large-scale benchmark for LLVM issue resolution, containing 423 real-world, validated tasks collected from the LLVM project. We further develop LLVM-Gym, a scalable evaluation platform that automates issue reproduction, patch application, compiler building, and test execution. Using LLVM-Bench and LLVM-Gym, we conduct a comprehensive study of four representative LLMs, six retrieval configurations, and three agents. Our results show that current LLM-based issue resolution techniques remain limited on LLVM-Bench, with patch invalidity and build failures as the dominant failure modes. We further reveal a strong complementarity among different LLMs and agents, motivating LLVM-Ens, a lightweight ensemble approach that expands the patch space through integrating the patches generated by diverse techniques, filters incorrect and redundant candidates, and identifies the most promising solution. Our results show that LLVM-Ens achieves a resolution rate of up to 21.99%, further improving LLVM issue resolution.

• The paper introduces LLVM-Bench, a large-scale benchmark evaluating LLM approaches on challenging LLVM compiler issues.
• Methodologies include extensive filtering, static and dynamic validations, and the automated LLVM-Gym environment.
• Results reveal low overall resolution rates and demonstrate ensemble methods (LLVM-Ens) that significantly boost performance.

LLVM-Bench: A Large-Scale Benchmark for LLM-Based LLVM Compiler Issue Resolution

Introduction and Motivation

LLVM, as a pervasive compiler infrastructure, represents the epitome of large-scale, complex, and evolving system software. Its massive and modular C/C++ codebase, intricate component interaction, and continuous integration requirements pose unique challenges to both manual and automated issue resolution. While LLM-based agents and retrieval-augmented methods have achieved notable progress on application-centric benchmarks such as SWE-bench, their efficacy on compilers like LLVM, which exhibit orders of magnitude higher complexity and domain specificity, remains uncharted. Addressing this crucial gap, "LLVM-Bench: Benchmarking and Advancing LLMs for LLVM Compiler Issue Resolution" (2607.00700) introduces LLVM-Bench, an empirically rigorous benchmark and pipeline for comprehensive evaluation of LLM-driven issue resolution in LLVM. The study further presents LLVM-Gym—an automated evaluation environment—and proposes the ensemble technique LLVM-Ens informed by systemic cross-technique complementarity observed during extensive experiments.

Benchmark Design and Construction Pipeline

LLVM-Bench targets repository-level LLVM issues resolved via merged pull requests with coupled test cases, maximizing real-world relevance and reproducibility. Its construction proceeds via multi-stage filtering, integrating static and dynamic validation and meticulous manual curation.

Figure 1: The pipeline of benchmark construction, spanning commit mining, automated validation, and manual curation to yield 423 high-quality LLVM issue resolution instances.

Starting from over 561,000 commits, the collection focuses on four major versions, isolates issues explicitly linked to merged PRs with new or modified tests, and partitions code changes into “golden patches” and test patches. LLVM-Gym conducts multi-stage automated validation: issues are reproducible if patched tests fail on buggy commits and succeed when the golden patch is added. Subsequent manual inspection guarantees unambiguous descriptions, absence of solution leakage, and proper categorization. The resulting benchmark consists of 423 issues, dominated by bug fixes (75.41%) and spanning all LLVM compiler components.

LLVM-Bench tasks dwarf prior benchmarks in scale: codebase sizes exceed 149k files and 47M LOC, golden patches average 130 lines, and issues require upwards of 72 days and several discussion rounds for manual resolution, confirming their non-triviality.

Systematic Study: LLMs, Agents, Retrieval, and Error Analyses

Evaluation Protocol

The paper conducts a factorial study of four representative LLMs (DeepSeek, Qwen, Gemini, Grok), six retrieval settings (Sparse/Oracle, three context lengths), and three advanced agent frameworks (SWE-agent, Trae-agent, Live-SWE-agent)—yielding 36 approaches. Effectiveness is measured via: %Applied (syntactic patch validity), %Built (compilability), %Resolved (all tests pass). Efficiency is quantified by token consumption and dollar cost.

Main Findings: Single Model/Agent Limits

Across all LLMs, even with privileged Oracle Retrieval, %Resolved remains below 5%. Modern agent-based techniques surpass pure LLM+retrieval mechanisms, with the best agent (SWE-agent with Grok) achieving 10.87% %Resolved. Aggregate analyses show clear complementarity (union “oracle” performance exceeds 45%), but no single method approaches the efficacy required for practical automation. Larger retrieval contexts bolster effectiveness but induce significant cost increases, highlighting the nontrivial context utilization and summarization bottleneck.

Error/failure Mode Analysis

A granular breakdown reveals that:

• Patch invalidity (~64%) and build failures (~82% of applicative patches) are the overwhelming failure modes, not functional errors.
• Unresolved tasks are reliably harder for both LLMs/agents and humans, requiring more edits and developer effort.
• Agents and LLMs perform substantially better on bug fixes versus new-feature tasks; component and version appear less significant.
• Models tend to resolve disjoint sets of issues, demonstrating strong complementarity potential for ensemble approaches.

  Figure 2: Number of uniquely resolved issues across different LLMs and agents, demonstrating substantial complementarity and non-overlapping solution spaces.

  

  Figure 3: Funnel chart of failure reasons for incorrect patches, illustrating the predominance of patch invalidity and build failures over localization or functional errors.

Ensemble Resolution: LLVM-Ens

Motivated by complementary resolution sets across agents/LLMs, LLVM-Ens leverages a three-stage pipeline: (1) dynamic patch filtering (via LLVM-Gym), (2) code normalization and de-duplication, and (3) LLM-based ensemble selection. This boosts %Resolved to 21.99%, a relative improvement of 339% over the best individual strategy. Importantly, LLVM-Ens’s gains arise not merely from larger candidate pools but from effective selection among diverse, non-redundant candidates.

Practical and Theoretical Implications

For Benchmarking

LLVM-Bench exposes that state-of-the-art issue resolution agents, which exceed 80% on application-level benchmarks, plateau below 11% for LLVM, revealing a significant generalization and scaling challenge for current LLM paradigms in system software engineering.

For Model and Agent Development

• Context Handling: Mere expansion of context window offers diminishing returns; context summarization, task-specific retrieval, and domain-constraint induction are critical research targets.
• Failure Mitigation: Patch syntheses should integrate both static (e.g., ClangFormat, Clang-Tidy) and dynamic validation in an iterative loop, to address invalidity and build errors.
• Task-Aware Agents: Separate workflows for new-feature and bug-fix issues are required, as generalized agentic scripts underserve diverse LLVM problem types.

For Ensemble Methods

Complementarity-aware ensemble reasoning exploits model and agent diversity, but effective selection in large hypothesis spaces remains computationally expensive; optimal diversification is key for future ensemble frameworks.

Conclusion

LLVM-Bench and LLVM-Gym introduce a rigorous infrastructure for assessing the capabilities and limitations of LLM-based issue resolution in large-scale system software, shifting the evaluation paradigm from application-centric to compiler-grade challenges. Empirical study demonstrates current SOTA approaches remain practically insufficient, with pronounced limitations in patch validity, build reasoning, and new-feature synthesis. The proposed LLVM-Ens ensemble method delivers significant gains by harnessing cross-approach complementarity, yet further advances demand architectural, representational, and agentic refinements tailored for system-scale repositories. LLVM-Bench thus constitutes both a call to arms and a foundational resource for advancing automated program repair, synthesis, and understanding in the most demanding real-world compiler contexts.

---

• Full paper: "LLVM-Bench: Benchmarking and Advancing LLMs for LLVM Compiler Issue Resolution" (2607.00700)
• Additional enumerated citations are in the document text.