On the Role of Fault Localization Context for LLM-Based Program Repair

Abstract: Fault Localization (FL) is a key component of LLM-based Automated Program Repair (APR), yet its impact remains underexplored. In particular, it is unclear how much localization is needed, whether additional context beyond the predicted buggy location is beneficial, and how such context should be retrieved. We conduct a large-scale empirical study on 500 SWE-bench Verified instances using GPT-5-mini, evaluating 61 configurations that vary file-level, element-level, and line-level context. Our results show that more context does not consistently improve repair performance. File-level localization is the dominant factor, yielding a 15-17x improvement over a no-file baseline. Expanding file context is often associated with improved performance, with successful repairs most commonly observed in configurations with approximately 6-10 relevant files. Element-level context expansion provides conditional gains that depend strongly on the file context quality, while line-level context expansion frequently degrades performance due to noise amplification. LLM-based retrieval generally outperforms structural heuristics while using fewer files and tokens. Overall, the most effective FL context strategy typically combines a broad semantic understanding at higher abstraction levels with precise line-level localization. These findings challenge our assumption that increasing the localization context uniformly improves APR, and provide practical guidance for designing LLM-based FL strategies.

• The paper demonstrates that incorporating file-level context can improve repair success by 15–17x compared to a baseline.
• LLM-based semantic retrieval at file and element levels reduces cost and outperforms traditional heuristic methods, as shown by statistically significant results.
• Excessive context at the line level may degrade performance, underscoring the need for adaptive, ensemble context strategies in automated program repair.

The Impact of Fault Localization Context on LLM-Based Automated Program Repair

Introduction

The paper "On the Role of Fault Localization Context for LLM-Based Program Repair" (2604.05481) presents a large-scale empirical investigation into how the granularity and strategy of fault localization (FL) context selection affect the effectiveness of LLM-based automated program repair (APR). The central research question is: how much and what kind of localization context maximizes LLM-based program repair success?

Systematically analyzing 61 context configurations on 500 SWE-bench Verified benchmark instances using GPT-5-mini, the authors provide a rigorous quantitative and qualitative dissection of file-, element-, and line-level FL granularity, and the comparative efficacy of heuristic, structural, and LLM-based semantic context retrieval. The results force a reassessment of several common assumptions in LLM-based APR regarding the monotonic benefits of localization precision, the utility of context expansion, and the interaction of abstraction levels in context provisioning.

Experimental Design

The study is defined by a factorial experiment that orthogonally varies the context at three levels:

• File-Level: Context options include none, “buggy” only, rule-based relevant (import graph expansion), and LLM-retrieved relevant files.
• Element-Level: Options span none, “buggy only”, Call Graph expansion, and LLM-based retrieval.
• Line-Level: Configurations include none, “buggy” only, ±10 line window, static slicing, and LLM-retrieved lines.

The key experimental control is the provision of ground-truth locations (files/elements/lines touched by developer patches) to isolate the impact of context selection from FL accuracy. Each configuration is embedded in a standardized two-tier prompt architecture, carefully controlling for confounds tied to system instructions, user prompt content, and patch formatting constraints.

The repair model receives a single repair attempt per context configuration on each instance. The principal evaluation metric is the number of instances successfully repaired (passing all validation tests).

Principal Findings and Numerical Results

File-Level Context Dominates APR Success

• Merely introducing file-level context yields a 15–17x improvement over the no-file baseline (from 3.6% to 56–63% resolution rate).
• Both rule-based (import graph) and LLM-based file expansion show significant improvement over buggy-only files. LLM-guided retrieval is consistently superior (statistically significant, Wilcoxon $p<10^{-4}$, large effect size), with the optimal context typically requiring 6–10 files.

LLM-Based Retrieval: Efficient and Effective Context Expansion

• LLM-based retrieval incurs lower cost in file and token budgets (average 8.54 files and ~58K tokens) relative to rule-based methods (~18 files, ~96K tokens), with significant cost reduction verified by Wilcoxon tests ($p \ll 0.01$).
• The LLM-retrieval approach maintains higher precision, especially when the editing location is well localized at the line level.

Element-Level Context: Conditional Utility

• Adding element-level context improves performance over “none,” but expansion beyond buggy elements (e.g., Call Graph expansion) is not consistently beneficial—structural expansion provides negligible or adverse returns.
• LLM-based element retrieval outperforms Call Graph-based retrieval (Wilcoxon $p=0.0366$, moderate effect); the best results are realized when pairing broad semantic element context with precise line localization.

Line-Level Context: Risk of Noise Amplification

• Expanding line-level context beyond precise buggy lines frequently degrades performance. Context windows and static slicing methods tend to amplify noise and distract from true localization.
• LLM-retrieved lines outperform structural expansion, but even these gains are modest and only materialize when file/element context is semantically relevant.

Interaction Effects and Best Configurations

• Optimal repair is realized by combining broad, semantically guided file/element context with precise line-level localization. The best configuration—LLM-retrieved files, LLM-retrieved elements, and buggy lines—achieves a 63.4% success rate, outperforming all uniform (fully precise or fully expanded) strategies.

Adaptive/Ensemble Context Strategies

• No single context configuration resolves all bugs; performance is complementary across configurations.
• Qualitative error analysis demonstrates that excessive or irrelevant context can shift the model’s repair reasoning toward over-generalization or architectural refactoring rather than localized patching.
• Conversely, some failures are due to missing cross-component dependencies that are only revealed via context expansion.
• An adaptive or ensemble context selection strategy, dynamically scaling context granularity per instance, is required for robust APR.

Implications

Practical Guidance for LLM-Based APR System Design

• Emphasize accurate file-level localization—returns on investment at the file level vastly exceed those for finer-grained expansion.
• Prefer LLM-based semantic retrieval over heuristic/structural expansion at both the file and element abstraction levels, balancing cost and performance.
• Keep line-level context focused—minimize expansion, as imprecision at this granularity has a disproportionately negative impact on repair rates.
• Develop adaptive or ensemble retrieval strategies—utilize bug characteristics to modulate context scope, leveraging complementary strengths across localization schemes.

Theoretical Implications

The study challenges the traditional FL-APR paradigm that pushes for monotonic improvement via increased context precision or coverage. The results demonstrate that:

• The utility of context is highly non-uniform and granularity-sensitive.
• Excessive expansion can be as detrimental as under-provisioning, especially at the line level.
• Semantic (intent-aware) context retrieval is critical for LLM reasoning, outperforming heuristic/structural selection.

These conclusions reorient fault localization research toward LLM-aware semantic retrieval, precision-focused line localization, and hybrid strategies that exploit complementarity between broad program understanding and focused editing.

Speculation on Future AI Developments

Future directions indicated by this research include:

• Model-driven FL systems: Developing LLMs dedicated to semantic context ranking at multiple granularities, optimizing for repair utility—not just localization accuracy.
• Meta-prompting and agentic repair systems: Integrating adaptive, iterative localization agents that modulate context scope based on early repair signals or observed model uncertainty.
• Cross-lingual/generalization studies: Evaluating whether the observed principles hold across other programming languages, LLM architectures, and APR benchmarks.
• Automated context calibration: Training models to predict the optimal context configuration using repository features, natural language bug descriptions, or code meta-data.

Conclusion

This paper provides an authoritative empirical characterization of how fault localization context influences LLM-based APR. The dominant effect of file-level context, the conditional value of element context, and the risk of noise at the line level are rigorously quantified. LLM-based semantic retrieval emerges as a robust, cost-efficient approach, but context strategy must be adaptive rather than rigid.

These findings demand a nuanced rethinking of APR pipeline design, with direct implications for LLM-powered engineering systems, agentic repair architectures, and the broader automation of software maintenance. Future innovations in context engineering, influenced by this study, are poised to substantially improve the scope and effectiveness of LLM-driven program repair.