An Evaluation of Role-Based Multi-Agent Code Generation on Repository-Scale Problems

Abstract: Role-based multiagent code generation aims to make LLMs more effective on repository-scale problems, moving beyond small programming tasks. We evaluate this approach on 12 Java repositories, finding greater similarity to developer code than single LLMs, but a persistent gap from human implementations.

• The paper demonstrates that reflexive multi-agent workflows consistently achieve higher code similarity and scenario-intent alignment than standalone LLM approaches.
• The study outlines a role-based pipeline that decomposes requirements into planning, coding, and iterative assessment phases, resulting in structurally robust code versus conventional methods.
• The evaluation highlights a trade-off where increased resource consumption in agentic pipelines leads to improved code scaffolding but still falls short of generating fully executable systems.

Evaluation of Role-Based Multi-Agent Code Generation at Repository Scale

Introduction

This paper, "An Evaluation of Role-Based Multi-Agent Code Generation on Repository-Scale Problems" (2607.04212), systematically investigates the capabilities and limitations of distributed, role-based LLM agent workflows for synthesizing Java repositories from structured requirements. The work benchmarks both a sequential and a reflexive multi-agent orchestration pipeline—decomposing requirements, assigning specialized agent roles, generating code, and iteratively reflecting on requirement compliance—against a standalone LLM baseline on 12 Java repositories of varying complexity. Using metrics evaluating syntactic and semantic similarity, scenario-intent representation, compilation success, and resource consumption, the study provides detailed empirical insights into the strengths and persistent challenges of agentic code generation at realistic repository scales.

Agentic Code Generation Pipeline

Traditional LLM-based code synthesis excels on isolated functions or small modules, but falters with full-scale software repositories exhibiting architectural layering, cross-file dependencies, and constraints requiring global consistency. The paper implements a multi-phase, agentic generation pipeline:

1. Planning: An LLM agent parses the input SRS (adhering to IEEE 29148) to extract tasks, assigns each to agents in dynamically determined specialist roles (planner, developer, tester, etc.), and sequences execution.
2. Coding: Each agent, assuming a prescribed role, synthesizes code fragments for their assigned task using production-oriented prompts, passing succinct summaries of changes to downstream agents for context synchronization.
3. Assessment (for reflexive workflows): An assessment agent evaluates the generated code against SRS requirements, identifying omissions/mismatches and triggering planning/coding refinement loops.
4. Setup: The system finalizes the codebase—including Maven-specific configuration—for compilation and build.

   Figure 1: Logical architecture of Agentic Code Generation, illustrating planning, coding, assessment, and setup phases with role-specific LLM agents communicating iteratively.

The reflexive variant incorporates iterative requirement-level reflection and repair, while the sequential variant proceeds linearly from plan to code.

Experimental Methodology

The evaluation targets 12 post-training-set Java repositories sampled across four complexity strata, ensuring minimal data leakage from model training. High-quality SRS documents were programmatically derived and manually validated from README and Javadoc sources, distilling implementation-agnostic requirements.

For each repository, the study compares:

• Standalone LLM: Direct code generation from SRS in a single prompt.
• Agentic Sequential: Multi-agent task decomposition without assessment/reflection.
• Agentic Reflexive: Iterative reflection on requirement fulfillment, re-planning until no issues are detected or a fixed limit is reached.

Metrics for evaluation included process statistics (number of agents/tasks), codebase size (LOC, classes, methods), syntactic similarity (JPlag), semantic/dataflow similarity (CrystalBLEU), scenario-intent matching (LLM-as-a-judge with Gherkin scenario extraction), compilation rate, and resource cost (time, tokens, monetary).

Results and Analysis

Process and Code Structure

Agentic workflows (both sequential and reflexive) produced well-structured, layer-separated code resembling standard domain-driven architectures. The number of planning tasks and agent invocations scaled with repository complexity. Inter-agent communication summaries achieved high coverage of modified artifacts (84–91%), validating this strategy for limited-context LLM pipelines.

Code Size and Compilation

Agentic codebases were smaller than those of the standalone LLM but captured key abstractions. Manual review confirmed that agentic outputs provided robust scaffolds: clear layering and core domain features, albeit lacking in architectural glue (e.g., adapters, resolvers) and complex cross-cutting concerns (security, error handling). Notably, the proportion of directly compilable classes was highest for the reflexive agentic workflow, though overall compilation rates were low—underscoring the bottleneck induced by the absence of compiler- or test-in-the-loop feedback.

Similarity and Scenario-Intent Representation

Agentic approaches consistently outperformed the standalone LLM in code similarity (JPlag, CrystalBLEU) and semantic scenario-intent metrics, particularly on moderate and complex repositories. The reflexive workflow surpassed the sequential agentic and LLM-only pipelines in aligning generated behaviors with SRS-derived Gherkin scenarios. Similarity metrics exhibited reduced per-project variance for reflexive workflows, evidencing convergence towards stable solutions through iterative, requirement-oriented repair.

Resource Consumption

Agentic approaches incurred greater computational cost and latency than the standalone LLM—up to $1.40 per complex repository, versus$0.28 for standalone LLM. However, these represent tractable resource investments for prototyping, given the qualitative improvements in codebase fidelity and intent capture.

Strong Empirical Findings

• Reflexive, role-based multi-agent pipelines repeatedly achieved higher code similarity and scenario-intent coverage relative to standalone LLMs, especially at nontrivial repository scales.
• Despite improvements, all approaches produced only partial, non-executable codebases: compilation success remained limited, and substantive human intervention was indispensable for functionality beyond initial scaffold generation.
• Agentic designs can over-engineer solutions for small applications, introducing superfluous architectural detail.
• Current assessment phases, lacking dynamic compilation or test execution, are fundamentally limited in their capacity to close the gap towards fully functional systems.

Implications and Future Directions

The results highlight that agentic AI significantly narrows the fidelity gap between generated and developer-maintained repositories, structurally aligning codebases and representing more functional intent from SRS documents. However, without dynamic feedback loops—such as compiler error analysis or test-based validation—current pipelines cannot synthesize fully functioning systems.

Practically, agentic workflows are best positioned today as advanced, cost-effective prototyping aids, accelerating initial implementation and systematizing role-based code decomposition. Theoretically, this work motivates research into integrating compiler/test-in-the-loop reflection, richer agent role heterogeneity, more nuanced cross-agent communication, and scalable bench-marking beyond surrogate similarity metrics. Increasing the expressiveness and accuracy of requirement-to-code mapping, along with robust execution-aware repair, are necessary to bring agentic software synthesis closer to full automation.

Conclusion

This study provides a rigorous empirical assessment of role-based, multi-agent LLM code generation workflows at repository scale. Reflexive agentic approaches surpass standalone LLMs in code quality, structure, and alignment with requirements, yet remain constrained by static, requirement-level assessment and partial code generation. Agentic AI currently delivers superior scaffolding with realistic architectural proxies, but closing the remaining functionality and correctness gaps will require execution-aware, dynamic reflection and stronger integration of human-in-the-loop protocols. Fully automated, high-fidelity, executable repository synthesis remains an open challenge, but agentic orchestration establishes a substantive foundation for the next phase of program synthesis research.