Can Coding Agents Implement Missed Compiler Optimizations? Evaluating LLM Agents on LLVM Peephole Optimizations

Abstract: Coding agents built on LLMs are now capable of patching sizable real-world codebases, yet whether they can develop compiler optimizations remains an open question. To study this question, we introduce PeepholeBench, an evaluation framework whose tasks are constructed from real-world missed peephole optimizations reported against LLVM's InstCombine pass. Since missed peephole optimizations are typically fixed with small, localized patches, they offer a well-scoped but demanding testbed for coding agents: a correct fix demands rigorous reasoning about program semantics along with familiarity with optimizer-specific conventions. PeepholeBench derives its tasks from 21 resolved LLVM issues and 19 merged pull requests (PRs), supplies agents with only the issue context that existed before each fix, and assesses the resulting patches for both correctness and profitability. With PeepholeBench, we benchmark state-of-the-art coding agents on fixing missed peephole optimizations in LLVM's InstCombine pass, measuring their patches against the corresponding human-written fixes. We observe a tension between correctness and profitability, and no agent matches human developers on both dimensions at once. The dominant failure modes are overly narrow transformations and misuse of LLVM-specific mechanisms, errors that existing test suites rarely expose. Together, these results establish PeepholeBench as a realistic and challenging benchmark for coding agents, and suggest future directions for building agents that can more dependably assist compiler optimization development.

• The paper presents PeepholeBench, a framework to evaluate LLM-based coding agents' ability to implement missed LLVM InstCombine optimizations from real-world issues.
• It employs mutation-based test generation and semantic validation using Alive2 and llvm-mca to rigorously measure correctness and profitability.
• Empirical results reveal a trade-off: while agents generate buildable patches nearly 100% of the time, they fail to match human experts on combined semantic and performance improvements.

Evaluation of Coding Agents for Missed LLVM Compiler Optimizations via PeepholeBench

Introduction and Motivation

LLM-based coding agents have demonstrated competence in software maintenance and patching tasks, yet their capabilities in developing nontrivial compiler optimizations—particularly in the context of established toolchains such as LLVM—remains insufficiently characterized. Peephole optimizations in LLVM’s InstCombine pass exemplify tasks that are highly scoped but require precise semantic reasoning and domain-specific conventions. This work introduces PeepholeBench, a framework for systematically evaluating coding agents’ abilities to implement missed peephole optimizations in InstCombine, constructed from real-world LLVM issues and pull requests (PRs). The central research questions address agents’ abilities to synthesize correct and profitable optimizations, benchmarked directly against human expert patches.

PeepholeBench: Benchmark Design

PeepholeBench consists of tasks spanning 21 resolved LLVM InstCombine issues and 19 merged PRs, each restored to the code state preceding the human-generated fix. For every benchmark instance, the agent receives: (1) the context from the original GitHub issue, (2) alive2 example IRs, and (3) InstCombine development guidelines. Agents must synthesize patches without access to the human fix. The evaluation framework aggregates correctness and profitability using rigorous semantic validation, beyond simple test passing, and employs mutation-based test generation to systematically probe the generalizability and edge-case behavior of synthesized patches.

Figure 1: Overview of the PeepholeBench workflow.

Methodology

Task Formulation: Each task is specified as a patch needed for a missed InstCombine optimization using issue descriptions, Alive2 IR pairs, and customized prompts.

Mutation-Based Test Generation: Core test seeds are augmented by domain-specific IR mutation strategies (boundary value substitution, operand and type variation, flag changes, operator substitution, etc.) to yield substantial coverage of preconditions, over-generalization, and performance regressions.

Behavioral Validation: Patches are evaluated by (1) semantic correctness (using Alive2 to verify IR refinement under the LLVM memory/UB model) and (2) profitability (using llvm-mca to estimate changes in cycles and micro-ops, and to classify rewrites as profitable, neutral, mixed, or regressive). Importantly, passing standard regression tests is not considered sufficient for success under PeepholeBench.

Failure Mode Analysis: Representative agent failures are subjected to thematic analysis, with failure labels generated using multiple LLM-based annotators and further grouped with active human adjudication.

Empirical Evaluation

Experimental Setup

The evaluation bench contains 19 PRs (all merged post-September 2025, to preclude model data leakage), with each coding agent tested in three independent runs per task. Agents include Claude Code (Sonnet 4.6), Gemini CLI (Gemini 3 Flash), and Codex CLI (GPT 5.4 & 5.4 mini).

Strong Numerical Results and Headline Findings

• Buildability: All agents produce syntactically valid, buildable patches in nearly 100% of tasks.
• Correctness vs. Profitability Tension: No tested agent matches human expert patches in both dimensions simultaneously. Kodex CLI models (GPT 5.4) achieve the highest behavioral validity (up to 79% Pass@3); Gemini 3 Flash achieves the highest rate of profitable, non-regressive rewrites relative to human patches (42% Pass@3).
• Human Baseline: Strikingly, human patches do not achieve 100% behavioral validity, primarily due to profitability regressions, not semantic unsoundness.
• Generalization: Gemini 3 Flash uniquely discovers profitable optimizations in cases left uncovered by human patches (generalization rate: 3.7%).

Contradictory and Noteworthy Claims

• Human patches include nontrivial rate of performance regressions—nearly half of PRs include at least one regressive rewrite relative to the tested IR patterns.
• Seed regression tests both overconstrain and underconstrain evaluation: Syntactic checks (FileCheck) can reject more profitable rewrites, while test coverage gaps allow agents to pass with profitable regressions undetected.

Failure Analysis

Dominant Failure Modes:

• Under-generalization: Most agent failure cases arise from insufficient pattern generalization, failing to extrapolate from concrete issue examples to the full intended transformation family.
• LLVM-Specific Conventions: Incorrect handling of single-use ("one-use guard"), misapplication of poison/flag constraints, and improper fold insertion points result in pervasive correctness and performance failures.
• Non-Convergent Rewrites: Some agent patches introduce rewrite cycles, hanging the optimizer.
• Canonicalization and Profitability Mismatches: Variations in IR canonical forms affect backend lowering, exposing limitations in both benchmarking and agent modeling.

Empirical findings also underscore that current agentic coding workflows—especially those relying principally on syntactic test suites—are insufficient substitutes for deep semantic and performance validation in the context of compiler optimization development.

Implications and Theoretical Impact

The results motivate several future directions both for LLM-aided software engineering and compiler research:

• Tighter Integration of Semantic Validation and Feedback: Coding agents for optimization must invoke semantic/behavioral checkers (Alive2, llvm-mca) within their loop, not only for ex post validation, but potentially for learning and self-correction.
• Augmentation over Replacement: Agentic methods are most promising as tools for surfacing edge cases, proposing generalizations, and catching latent regressions in human patches—not as drop-in replacements.
• Representative Benchmarking: Mutation-based coverage is necessary for realistic measurement of agent generalization and profitability properties; syntactic or shallow regression tests are inadequate for evaluating real-world optimization competencies.
• Improved Codebase Awareness and Refactoring: Current agents tend to append new rules (increasing code size) rather than refactor existing ones for broader coverage and maintainability.

Practical Outlook

Further advances in LLM-agent capabilities will require: (1) explicit incorporation of domain-specific abstractions (pattern matching, flags), (2) more expressivity and coverage in mutation-based test generation, and (3) improved bridge mechanisms between semantic IR validation and model-driven code synthesis. Additionally, profitability measurement must be refined to capture not only local cycle reductions but broader pipeline and microarchitectural impacts, supporting prioritization of fixes by real-world significance. There is strong evidence to suggest that agentic co-design with human experts will become best practice for robust, maintainable compiler optimization development.

Conclusion

PeepholeBench offers a rigorous evaluation framework for LLM coding agents on real-world missed instcombine optimizations in LLVM, exposing a pronounced correctness-profitability trade-off that delineates current limits of agentic approaches. The failure profiles and empirical trade-offs detailed in this work provide concrete directions for both agent tooling and future research in automated compiler optimization (2607.02684).