The Illusion of Agentic Complexity in README.md Generation: Evaluating Single-Agent vs. Multi-Agent RAG Systems

Abstract: LLMs are increasingly utilized to automate several software engineering tasks, including code completion, code summarization, testing, and the generation of repository-level documentation. While Multi-Agent Systems (MAS) are often adopted to support such tasks under the premise that task decomposition improves performance, the impact of architectural complexity on practical efficiency remains under-examined. This study empirically evaluates Retrieval-Augmented Generation (RAG) dependent architectures for the generation of README files for GitHub repositories. In this work, we conducted a systematic comparison between a Single-Agent pipeline, a specialized MAS, and a developer-guided planning (DevPlan) variant, benchmarked against LARCH -- a state-of-the-art baseline -- and the original ground truth. Results indicate a critical architectural trade-off: the Single-Agent pipeline achieves lexical quality comparable to MAS while reducing token consumption by 86% and operating at twice the speed. In contrast, manual taxonomy analysis demonstrates that MAS achieves high structural consistency (98%), resolving formatting issues observed in single-agent approaches. Autonomous planning is identified as the primary pipeline bottleneck; incorporating lightweight developer-guided plans produces the highest overall documentation quality, surpassing all the analyzed configurations.

• The paper finds that single-agent RAG pipelines provide similar lexical quality to multi-agent systems while using significantly fewer computational resources.
• The study demonstrates that multi-agent systems, particularly with human-guided planning, achieve superior structural precision and comprehensive documentation.
• Empirical evaluations reveal clear trade-offs between efficiency and structural quality, suggesting that added agentic complexity should be justified by specific needs.

Evaluating Agentic Complexity in Automated README Generation

Introduction

This paper ("The Illusion of Agentic Complexity in README.md Generation: Evaluating Single-Agent vs. Multi-Agent RAG Systems" (2606.30524)) addresses the open question of whether increased architectural complexity—specifically, the use of multi-agent systems (MAS)—provides substantive advantages over monolithic single-agent retrieval-augmented generation (RAG) pipelines for automated repository-level documentation. Despite the prevailing intuition that modular decomposition and role-based collaboration in MAS promote higher quality and structural consistency, systematic empirical analyses for README.md generation have been absent from the literature. This work introduces a rigorous comparative framework integrating both algorithmic and human-guided planning, using ROUGE, BERTScore, and structural evaluation to benchmark performance across single-agent, MAS, and state-of-the-art baseline systems.

Generation Pipeline Design

The architecture of the experimental frameworks is detailed in Figure 1, which illustrates the modular pipeline, starting from repository ingestion and semantic indexing, and culminating in the generation and evaluation of documentation artifacts.

Figure 1: Overview of the proposed README generation pipeline, from repository preparation and indexing to generation and evaluation.

Repository preparation involves the extraction of canonical header patterns from ground-truth README files, structural isolation of documentation, and the construction of a section taxonomy guiding downstream queries. The semantic indexing pipeline employs language-aware chunking, static analysis to identify salient files, and the construction of repository-specific vector stores utilized for context retrieval.

The generation phase consists of three paradigms: (1) a single-agent RAG pipeline that leverages broad structural context in a monolithic prompt; (2) a MAS workflow featuring orchestration, profiling, outline planning, section-level drafting, and iterative review by specialized agents; and (3) a developer-guided MAS variant ("Dev-Plan"), which substitutes the automated planning with a human-crafted document outline, retaining the downstream MAS workflow.

Experimental Methodology

A filtered sample of 180 non-archived, non-fork repositories created after August 2025 serves to mitigate LLM data contamination. Metrics include ROUGE-L F1, BERTScore, LLM-based structural judgment, and manual taxonomy coverage. Token efficiency and latency are recorded to contextualize computational trade-offs. Notably, the original ground-truth README files are excluded from system input to ensure strict isolation of generative signals from pretrained LLM knowledge.

Empirical Findings

Quality and Efficiency Trade-offs

Quantitative results reveal that increased agentic complexity does not linearly translate to improved content quality. Figure 2 illustrates ROUGE and BERTScore results across single-agent, MAS, and LARCH baselines.

Figure 2: Comparison of ROUGE and BERTScore for MAS, LARCH, and Single-Agent systems.

Single-agent and multi-agent pipelines exhibit nearly identical ROUGE-L F1 and BERTScore (0.2007 vs. 0.1964, and negligible BERTScore differences), both significantly outperforming LARCH (ROUGE-L F1: 0.1022). However, efficiency diverges sharply: as depicted in Figure 3, MAS induces a 7.1x increase in token consumption and nearly doubles inference time relative to the single-agent configuration.

Figure 3: Token usage by MAS, LARCH, and Single-Agent methods.

In the Dev-Plan (human-planned MAS) configuration, structural quality increases (ROUGE-L F1: 0.2323, BERTScore F1: 0.8230), but at the cost of further computational overhead due to increased section-level granularity and repeated review cycles.

Structural Consistency and Organizational Quality

Manual taxonomy analysis and LLM-as-a-Judge evaluations demonstrate that MAS achieves superior structural precision (98.2%) and consistently low failure rates in documentation segmentation. Single-agent prompts, despite lexical coherence, are prone to organizational omissions and exhibit only 49.2% structural recall. Dev-Plan attains the highest structural recall (75.0%) and LLM-as-a-Judge scores (Mean: 8.60/10), confirming that planning constitutes the dominant bottleneck for fully automated MAS pipelines.

Statistical and Qualitative Insights

Statistical testing (Wilcoxon signed-rank with Holm-Bonferroni correction) confirms significant improvements in formatting and coverage for MAS and Dev-Plan over both monolithic LLM-based pipelines and human-authored README files. Notably, even exemplars of state-of-the-art heuristic single-file baselines (LARCH) underperform both RAG-based automated methods and MAS variants, due to an inability to holistically encode project structure.

Implications and Future Directions

The empirical evidence disrupts assumptions about the necessity of agentic complexity in repository-level documentation tasks. While MAS reliably enforces structural conventions and higher precision, the single-agent pipeline achieves analogous lexical quality at drastically reduced computational cost. However, both automated paradigms surpass human README authors in structural coverage, emphasizing the capability of these systems to internalize literature-driven documentation standards.

The bottleneck in MAS performance is localized to automated planning; limited capacity to synthesize project-specific outlines constrains downstream section quality. Human-in-the-loop interventions ("Dev-Plan") achieve superior outcomes, but are accompanied by increased resource consumption. This highlights the emerging principle: leveraging LLMs for execution while maintaining human oversight for planning or taxonomy design is currently optimal when maximal documentation comprehensiveness is required.

Further research is warranted to generalize these findings to more complex software engineering tasks, including automated test generation and bug fixation workflows, and to refine collaborative human-AI delegation strategies for large scale documentation synthesis.

Conclusion

This study demonstrates that, for README generation, single-agent RAG pipelines provide a highly efficient alternative to MAS, with negligible losses in content quality but significant advantages in resource utilization. MAS achieves superior structural uniformity and completeness—particularly with human planning—but at substantial computational expense. These findings have broad implications for the design of automated documentation systems in software engineering, and suggest that architectural complexity should be intentionally justified based on concrete structural or governance requirements rather than assumed benefit.