SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit in the Era of Autonomous Agents

Abstract: Although LLMs have demonstrated impressive coding capabilities, their ability to autonomously build production-scale software from explicit specifications remains an open question. We introduce SWE-AGI, an open-source benchmark for evaluating end-to-end, specification-driven construction of software systems written in MoonBit. SWE-AGI tasks require LLM-based agents to implement parsers, interpreters, binary decoders, and SAT solvers strictly from authoritative standards and RFCs under a fixed API scaffold. Each task involves implementing 1,000-10,000 lines of core logic, corresponding to weeks or months of engineering effort for an experienced human developer. By leveraging the nascent MoonBit ecosystem, SWE-AGI minimizes data leakage, forcing agents to rely on long-horizon architectural reasoning rather than code retrieval. Across frontier models, gpt-5.3-codex achieves the best overall performance (solving 19/22 tasks, 86.4%), outperforming claude-opus-4.6 (15/22, 68.2%), and kimi-2.5 exhibits the strongest performance among open-source models. Performance degrades sharply with increasing task difficulty, particularly on hard, specification-intensive systems. Behavioral analysis further reveals that as codebases scale, code reading, rather than writing, becomes the dominant bottleneck in AI-assisted development. Overall, while specification-driven autonomous software engineering is increasingly viable, substantial challenges remain before it can reliably support production-scale development.

• The paper introduces a new benchmark, SWE-AGI, that evaluates LLMs' ability to develop complete software systems from detailed specifications using MoonBit.
• It measures performance through 22 progressively challenging tasks, achieving an 86.4% success rate for advanced models on moderate complexity tasks.
• The study reveals a shift from code generation to comprehension as task complexity increases, highlighting current limitations in architectural reasoning.

SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit

Introduction to SWE-AGI Benchmark

The "SWE-AGI" benchmark addresses the critical need for an evaluation framework that encapsulates the challenges of end-to-end, specification-driven software development in the context of LLMs. Traditional benchmarks tend to emphasize isolated code completion tasks which fall short in assessing the full capabilities of LLMs in real-world software engineering scenarios. SWE-AGI shifts focus towards specification-driven software construction, evaluating an LLM's ability to implement complex software systems from scratch. Utilizing MoonBit, a modern programming language, this benchmark requires agents to develop systems solely based on detailed specifications with strict adherence to predefined API scaffolds.

Benchmark Structure and Task Design

SWE-AGI comprises 22 tasks spanning various categories such as parsers, interpreters, and SAT solvers, which represent significant engineering challenges, requiring implementation of 1,000–10,000 lines of core logic. These tasks are structured to emulate realistic software development processes, hence evaluated based on agents' ability to comprehensively understand and implement specifications without relying on retrieved code snippets. The framework reduces data leakage risks by leveraging MoonBit's nascent ecosystem, thereby emphasizing architectural reasoning over code retrieval. Moreover, SWE-AGI tasks are stratified by difficulty—easy, medium, and hard—based on code volume and complexity, posing challenges akin to several weeks or months of human engineering effort.

Evaluation of LLM-Based Agents

The benchmark assesses various LLMs in performing specification-driven tasks using MoonBit under controlled evaluations with deterministic outcomes. The leading model, gpt-5.3-codex, achieved a remarkable 86.4% success rate, solving 19 out of 22 tasks, showcasing superior performance over claude-opus-4.6 and kimi-2.5. However, as task complexity increased, particularly in specification-intensive systems, performance declined significantly. This decline underscores the gap in current LLM capabilities when tasked with maintaining architectural coherence over large codebases, where code reading emerges as a major bottleneck.

Behavioral Analysis and Implications

An in-depth analysis of LLM agent behaviors highlighted the growing dominance of code comprehension activities, surpassing code generation as task difficulty escalates. Specifically, as implementations scale, maintaining system integrity and understanding existing structures become central challenges. This finding aligns with industry observations that long-term software engineering is increasingly constrained by comprehensiveness and adaptability rather than outright code creation ability.

Implications for Autonomous Software Engineering

SWE-AGI provides important insights into the challenges of autonomous software engineering, particularly emphasizing the need for advancements in LLM capabilities to manage complex architectural reasoning. The benchmark suggests the feasibility of specification-driven software engineering becoming more attainable as LLMs evolve. Future work may explore multi-modal task inputs and integrate non-functional requirements such as maintainability and security, further bridging the gap towards fully autonomous, production-grade software engineering.

Conclusion

SWE-AGI sets the stage for future investigations into the capabilities of LLM-assisted software engineering, providing a robust framework for evaluating specification-driven system architecture and implementation tasks. It highlights both the potential and the obstacles in the path towards achieving autonomous software development, serving as a critical tool for both academic inquiry and practical application development.