SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration

Abstract: LLM-powered agents have demonstrated strong capabilities in automating software engineering tasks such as static bug fixing, as evidenced by benchmarks like SWE-bench. However, in the real world, the development of mature software is typically predicated on complex requirement changes and long-term feature iterations -- a process that static, one-shot repair paradigms fail to capture. To bridge this gap, we propose \textbf{SWE-CI}, the first repository-level benchmark built upon the Continuous Integration loop, aiming to shift the evaluation paradigm for code generation from static, short-term \textit{functional correctness} toward dynamic, long-term \textit{maintainability}. The benchmark comprises 100 tasks, each corresponding on average to an evolution history spanning 233 days and 71 consecutive commits in a real-world code repository. SWE-CI requires agents to systematically resolve these tasks through dozens of rounds of analysis and coding iterations. SWE-CI provides valuable insights into how well agents can sustain code quality throughout long-term evolution.

• The paper introduces an evolution-based evaluation loop that measures long-term maintainability and technical debt in codebases.
• It proposes a novel dual-agent CI cycle where an Architect and a Programmer collaboratively simulate real-world continuous integration.
• The study defines Normalized Change and EvoScore metrics, revealing that state-of-the-art LLMs still struggle with zero regression in extended maintenance cycles.

SWE-CI: A Benchmark for Long-Term Codebase Maintenance via Continuous Integration

Motivation and Benchmark Paradigm Shift

SWE-CI addresses a critical deficiency in the current evaluation landscape for code intelligence: a lack of metrics and protocols that reflect an agent's ability to maintain and evolve codebases over time, rather than simply optimizing for short-term, snapshot-based correctness. Previous benchmarks such as HumanEval, MBPP, and SWE-bench have evaluated LLM-powered agents using static inputs and single-shot solutions, obscuring the compounding effects of early design decisions on future maintainability. SWE-CI proposes a paradigm shift by introducing an evolution-based evaluation loop, making maintainability and the control of technical debt measurable attributes of agent performance.

Figure 1: Unlike previous benchmarks, SWE-CI proposes an evolution-based evaluation. The red and blue arrows represent the actions of functions $\mathsf{require}$ and $\mathsf{code}$, respectively. Dashed lines indicate processes that are unknown to the user.

Benchmark Construction and Dataset Curation

The dataset underlying SWE-CI is constructed via a rigorous four-stage filtering and augmentation process, beginning with over 4,900 real-world, permissively licensed Python repositories and distilling down to 100 challenging base/oracle codebase pairs, each representing significant evolutionary distance. For each sample, the transition spans on average 71 consecutive commits and 233 days of authentic development, involving at least 500 non-test lines changed. Automated Docker-based environment inference and a self-repair mechanism ensure reliable reproduction and execution of test suites across historical commits, while final filtering discards trivial or low-value pairs.

Figure 2: Data curation process of SWE-CI.

Each SWE-CI sample is shipped with the complete source for both endpoints, all intermediate tests, and a pre-built runtime, establishing a highly controlled setting for agent evaluation. Compared to prior benchmarks, SWE-CI exemplifies a dramatic increase in both scale and ecological validity for studying code evolution.

Dual-Agent CI Loop Protocol

SWE-CI introduces a dual-agent evaluation protocol that heuristically models industrial CI/CD processes. An Architect agent analyzes test failures, performs root cause diagnosis, and incrementally drafts high-level requirement documents, restricting each iteration to 1–5 of the most urgent behavioral contracts. A Programmer agent consumes these requirements and independently modifies the codebase to meet them, operating strictly on production code and never altering tests. This protocol enforces clear separation of concerns and constrains regressive or circumventive behavior.

Figure 3: SWE-CI uses an architect-programmer dual-agent workflow to model the continuous integration cycle of professional software teams in the real world.

Each round, the functional gap is recomputed between the evolving codebase and the oracle, with the next set of requirements derived afresh. This serial dependency ensures that earlier design and implementation choices exert persistent influence on subsequent iterations, exposing both technical debt accumulation and maintainability gaps.

Metrics: Normalized Change and EvoScore

To convert the series of states traversed during iterative maintenance into a scalable and robust numerical evaluation, SWE-CI introduces two principal metrics:

• Normalized Change ($a(c)$): Quantifies improvement or regression for a codebase $c$ relative to its baseline and oracle endpoints, on a continuous $[-1, 1]$ scale. Symmetric normalization sharply distinguishes between advances and destructive regressions.
• EvoScore ($e$): A future-weighted mean of normalized changes over N iterations, parameterized by $\gamma$. For $\gamma > 1$, greater weight is assigned to later iterations, thereby prioritizing sustained long-term maintainability over immediate correctness. EvoScore enables fine-grained attribution of model strengths: aggressive, brittle short-term fixes are penalized in favor of well-factored, robust solutions carrying through extended evolution.

Experimental Evaluation and Key Results

Across 18 models from 8 providers, extensive experimentation (10B+ tokens) demonstrates rapid annual improvements in code maintenance abilities among leading LLMs. Within each provider family, newer generational models show monotonic EvoScore increases, with substantial performance boosts for post-2026 releases. Notably, the Claude Opus series consistently outperforms contemporaries across the entire observation window, with GLM-5 also exhibiting competitive results.

Figure 4: The EvoScore variation of state-of-the-art models from 8 providers on SWE-CI.

Ablation across the $\gamma$ parameter reveals that provider-level training or alignment strategies modulate preference for short-term gains versus long-term maintainability. Providers such as MiniMax, DeepSeek, and OpenAI GPT models exhibit long-term optimization, while others like Kimi and GLM exhibit a bias towards immediate returns. Claude and Qwen models maintain robust rankings regardless of $\gamma$, suggesting more stable internal optimization or more balanced training.

Figure 5: The model's EvoScore ranking changes as $\gamma$ increases. When $\gamma > 1$, higher-ranking models indicate better codebase maintenance.

Despite improving trends, most SOTA models achieve zero-regression rates below 0.25. Only the latest Claude Opus variants surpass 0.5, while the majority introduce regressions with high frequency during extended CI cycles. This exposes a pronounced gap between current LLM agent abilities on static benchmarks and the requirements of fully automated, long-lived code maintenance in real-world scenarios.

Figure 6: All models are sorted from smallest to largest by the zero regression rate.

Implications and Future Directions

The results presented by SWE-CI indicate that while LLMs are quickly closing the performance gap on static bug-fixing and snapshot-style code synthesis benchmarks, they remain deficient in maintaining code quality and avoiding regressions during long-term evolution. The divergence between functional correctness and maintainability becomes quantitatively apparent only in this new evaluation setting. SWE-CI exposes long-term planning capabilities, technical debt amortization, and an ability to incrementally reason about architectural and interface stability—factors essential for autonomous software agents operating in realistic environments.

The practical implication is immediate: deployment of LLM-powered agents in industrial CI/CD pipelines demands not only high snapshot accuracy, but robust maintenance with low regression risk over iterative cycles. Theoretically, SWE-CI invites further research into agentic planning, local and global code refactoring, test-aware prompting, and continual learning strategies capable of preserving modularity and interface contracts.

As public datasets and protocols like SWE-CI mature, one can anticipate that LLM and agent architectures will increasingly optimize for lifecycle-aware behaviors, and novel training paradigms leveraging maintenance history will proliferate. Ultimately, SWE-CI paves the way for automated agents capable of participating in genuine, multi-iteration, production-grade software development and maintenance.

Conclusion

SWE-CI proposes an essential evolution in benchmarking code intelligence: from static, single-round correctness to dynamic, continuous integration-based evaluation of long-term maintainability. Through rigorous agent-based protocols, reproducible curation of real-world evolutionary data, and carefully calibrated metrics, SWE-CI foregrounds the unresolved challenges in building truly robust software agents. As the SWE-CI benchmark is adopted, it provides a critical lens for both measuring and advancing the development of AI agents capable of sustaining high-quality software systems over extended lifecycles (2603.03823).