- The paper demonstrates that increasing AI autonomy leads to an 84% reduction in development hours while maintaining or improving requirements coverage.
- The paper reveals that agentic platform architecture, comparing GitHub Copilot and AWS Kiro, independently influences prompt efficiency and overall productivity.
- The paper finds a trade-off where reduced cognitive load comes with increased developer frustration, highlighting key human factors in agentic AI workflows.
Three-Phase Evaluation of AI-Assisted Software Development Life Cycle
Experimental Design and Evaluation Methodology
This study systematically evaluates the impact of escalating AI autonomy and selection of agentic platforms (GitHub Copilot and AWS Kiro) on software development productivity, requirements adherence, and developer cognitive load. The methodology consists of three sequential phases, each involving a complete reimplementation of the same full-stack web application by a stable team of four senior student developers. The phases correspond to (1) partial AI assistance (Copilot), (2) fully agentic AI workflow using Copilot, and (3) fully agentic AI workflow with AWS Kiro. Key evaluation metrics included total development hours, requirements coverage via a Requirements-to-Implementation Traceability Matrix (RITM), AI interaction efficiency, and NASA-TLX-based cognitive workload indicators.
The task uniformity and consistent developer team eliminate inter-project and inter-team variance but introduce learning/familiarity confounds, as all phases are conducted sequentially on the identical application. All metrics are derived from descriptive means (n=4) due to the pilot scale. Human-in-the-loop metrics (prompt success, time tracking) were subject to inter-rater consistency checks but not independent audit or statistical hypothesis testing.
Quantitative and Qualitative Findings
Development Efficiency
A monotonic 84% reduction in development hours is observed with each increase in AI autonomy (Phase 1: 36.0h, Phase 2: 9.4h, Phase 3: 5.9h), exceeding previously reported efficiency gains for LLM-tooling in code completion and extending them into agentic, nearly human-out-of-the-loop settings. This reduction is non-trivial and demonstrates that full agentic workflows can compound efficiency beyond partial-assistive baselines.
Code Quality and Requirement Adherence
Despite concerns that increased AI autonomy could erode requirement fidelity, RITM scores improved across phases (18, 19, 20 out of 22 for Phases 1-3, respectively). Notably, the highest requirement adherence coincided with the lowest human coding involvement, a finding at odds with the presumption that active human coding and review are prerequisites for requirement traceability. However, RITM captures only implemented requirements, not latent software quality (e.g., security, maintainability).
AI-Interaction Efficiency
Prompt success rates followed the same upward trend: Copilot partial (79.9%), Copilot agentic (85.0%), Kiro (96.0%). Kiro’s architecture—requiring more granular, spec-driven prompts—resulted in a slightly higher prompt-to-feature ratio (2.42) relative to Copilot agentic mode (1.92), yet with substantially higher per-prompt yield. These findings indicate that platform architecture, not just autonomy level, is an independent determinant of developer efficiency and interaction efficacy.
Developer Cognitive Load
NASA-TLX indices for effort and mental demand decreased with increasing AI autonomy (effort: 12.5→7.75→5.75; mental: 12.75→8.5→7.5). Temporal demand followed a comparable decline. However, frustration increased non-monotonically (6.25→6.75→7.5), coincident with reduced code authorship and greater agent orchestration. This divergence between cognitive relief and affective cost signals a specific class of human factors risks in agentic workflows—potentially arising from diminished agency, unfamiliar tooling, or increased reliance on prompt engineering.
Self-rated performance remained high and stable for all phases (16.25–17.5), and physical load was negligible—indicating no perceived diminishment in individual impact despite transition from manual development to agentic orchestration.
Implications and Theoretical Significance
Platform Architecture as a Decisive Factor
The 37% additional time savings observed between Copilot and Kiro at identical autonomy levels, combined with Kiro’s superior prompt yields and requirement coverage, provide strong preliminary evidence that agentic platform architecture contributes independently to development outcomes. This result underscores the necessity for comparative empirical evaluation of AI tooling rather than treating agentic AI as a commodity resource.
Cognitive and Role Transformation
Reduced cognitive workload in agentic phases suggests that human developer activity is recentered on higher-leverage tasks: requirements scoping, prompt design, and validation, aligning with current OODA-like models of agentic human-AI collaboration. The rise in frustration, however, demonstrates nuanced adoption friction. This cost, possibly rooted in reduced coding agency or tool unfamiliarity, has implications for onboarding, organizational change management, and the redesign of engineering roles in agentic settings.
Quality/Efficiency Tradeoffs in Well-Scoped Contexts
The absence of a negative correlation between efficiency and requirement adherence in this greenfield setting challenges prevalent safety arguments for continuous human-in-the-loop code review within controllable scopes. However, the study design does not address security, maintainability, technical debt, or complex system interaction effects, which limits external validity.
Future Research Directions
Methodologically, randomized parallelized designs, inclusion of professional developer cohorts, legacy and large-scale systems, and longer time horizons are required to isolate causal relationships and generalize findings. Comprehensive evaluative frameworks should integrate static/dynamic analysis, code quality metrics, defect rates, and longitudinal maintenance outcomes. Comparative studies across a broader space of agentic architectures and orchestration patterns are warranted to inform both organizational adoption and platform design.
Limitations
Key limitations include the fixed, sequential, non-counterbalanced design with a single student team, potential cross-phase learning effects, and reliance on self-reported/interpretable metrics. RITM narrowly measures requirement traceability; NASA-TLX responses may be influenced by retrospective bias; tooling versions and configurations were not systematically archived, which restricts reproducibility.
Conclusion
This three-phase empirical study demonstrates that increased AI autonomy and deliberate selection of agentic platform architectures can drastically reduce development time and, under controlled conditions, incrementally improve requirement adherence and lower cognitive workloads. However, full agentic workflows may introduce new affective burdens—specifically, increased developer frustration—that must be addressed through organizational support and role adaptation. While the findings are suggestive rather than definitive due to confounds and sample limitations, they provide a methodological foundation and set a research agenda for rigorous, scalable evaluation of agentic AI platforms in software engineering.