On the Viability of Requirements Generation From Code: An Experience Report

Abstract: Empirical research in Requirements Engineering is hampered by a lack of adequate datasets that pair source code with corresponding requirements. A tempting route to addressing this lack is the use of LLMs to synthesize requirements from existing code bases. We investigate this question by evaluating an LLM-based and RAG-supported agentic approach that generates requirements from source code, verifies their implementation status relying on a human-in-the-loop, and synthetically introduces requirements smells and non-implemented requirements. Our goal was to create datasets that mimic reality and foster empirical RE research. However, during the study, various problems arose, leading to this experience report. Contrary to our initial hypotheses, LLMs were unable to (i) generate non-implemented requirements reliably, (ii) generate high quality requirements, and (iii) reliably introduce synthetic requirements smells. Furthermore, neither an LLM nor a single human-in-the-loop suffices to detect requirements smells reliably. These findings suggest that the generation of code-to-requirements datasets using LLMs is not yet viable and requires human supervision, especially for quality assurance. We critically reflect on our lessons learned and draw relevant conclusions for both researchers and practitioners.

• The paper presents an empirical evaluation of an agentic, RAG-supported, human-in-the-loop pipeline that generates requirements from code.
• The paper finds that while LLMs achieve high precision in identifying implemented requirements, their recall and handling of synthetic requirements smells remain limited and subjective.
• The paper concludes that automated code-to-requirements generation is not yet viable without significant human validation, especially when dealing with varied code quality.

Agentic LLMs for Requirements Generation From Code: An Empirical Evaluation

Motivation and Problem Statement

Empirical Requirements Engineering (RE) research necessitates high-quality datasets pairing requirements with trace links to code. Existing datasets are often inadequate: industry datasets are largely inaccessible due to legal constraints, while academic datasets are typically limited in scale, quality, and realism. Recent advances in LLMs have prompted optimism that they might generate requirements from code, supplemented by synthetic artifacts such as requirements smells and non-implemented requirements, yielding datasets with characteristics closer to real-world scenarios. This paper critically evaluates an agentic, Retrieval-Augmented Generation (RAG)-supported human-in-the-loop pipeline for requirements generation from code, including the generation of synthetic requirements smells.

Approach Overview

The proposed architecture leverages generator, verifier, mutation, and trace-link recovery agents, each using LLMs grounded via RAG on code artifacts to ensure better traceability and artifact realism. The generator agent produces requirements from code shards, aiming for diversity and coverage; a subset of requirements are intentionally crafted as non-implemented (control group), and another subset mutated to introduce explicit requirements smells. Human evaluators (PhD students in RE) validate generated requirements for appropriateness, implementation status, and smells, with trace-link recovery facilitated by a specialized agent.

Multi-shot prompt feedback enables iterative adaptation, feeding human labels back into subsequent LLM generations. Evaluation involved two disparate projects—one a student software project (SEP), the other a mature open-source client (Mattermost)—to assess the approach's robustness across varying code quality.

Empirical Results

Implementation and Hallucination

LLM-generated requirements show high precision (0.857–1.000) but moderate recall (0.708–0.769), indicating reliable identification of implemented requirements but unreliable generation of true non-implemented artifacts. The hallucination rate is low overall (6.1%), but varies by codebase (10.3% SEP vs. 1.8% Mattermost), evidencing code quality dependence.

Requirements Quality: Smells and Subjectivity

24.5% of requirements labeled as smell-free by LLMs were peer-reviewed as containing requirements smells—mostly non-atomic, passive voice, ambiguities, negative statements, and subjective language. Inter-rater agreement for smell detection is weak (Cohen's k = 0.213), establishing that smells are highly context-dependent and not reliably identified even by experts without shared conventions.

Synthetic requirements smells generated by mutation agents are recognized as problematic in 90.9% of cases, but only 60.6% are categorized with exact-match agreement. Certain smells (e.g., loopholes, passive voice) are particularly challenging for human evaluators to detect or categorize.

Appropriateness and Perception

Appropriateness ratings are similarly subjective (Krippendorff's α = 0.178). Smelly requirements are consistently perceived as less appropriate, but the multi-shot feedback loop's impact is inconclusive due to limited batch counts.

Discussion and Lessons

The empirical findings contradict initial hypotheses: LLMs, even when grounded in code with sophisticated prompting, cannot reliably generate high-quality requirements, non-implemented requirements, or synthesize targeted requirements smells without substantial human oversight. Smell detection is inherently context-dependent, requiring consensus conventions and multiple evaluators.

Quality and reliability are strongly codebase-dependent: LLMs perform substantially better with well-structured, mature code (Mattermost) than in less rigorously developed student projects (SEP). Generated requirements frequently exhibit linguistic and structural deficiencies, even when functionally correct—a phenomenon also reported in related studies.

Practically, fully autonomous dataset generation is not viable. Generated requirements should only be used after human review. For research, the paper provides two peer-reviewed datasets, but warns against using them as ground truth without consideration of the inherent subjectivity and artifact limitations.

Implications and Future Directions

The study signals that existing agentic LLM approaches—even when supplemented by RAG and human-in-the-loop mechanisms—are not sufficient for autonomous requirements-from-code dataset creation. Human supervision remains indispensable, especially for quality assurance and artifact realism.

Theoretical implications include further evidence for the context-adaptive nature of requirements quality and defect detection, echoing previous work that requirements smells are not absolute but situated. Practically, improvements in prompt engineering or automated smell detection may modestly reduce human workload, but full automation is not foreseeable without advances in both LLMs and contextual grounding.

Future work should scale evaluations to broader domains, especially safety-critical systems, and explore hybrid workflows where automated smell detection preselects candidates for human verification, potentially streamlining quality assurance. Further exploration of the multi-shot feedback loop's adaptation capabilities is warranted.

Conclusion

Agentic, RAG-supported LLMs can assist requirements generation from code, but cannot supplant human evaluators for quality or realism. Requirements smells and appropriateness are highly subjective, and generated artifacts display substantial deficiencies irrespective of prompt engineering. The generalizability and reliability of the approach are constrained by codebase characteristics. Research and practice should remain cautious in adopting LLM-generated requirements artifacts, prioritizing human-in-the-loop validation and context-adaptive quality frameworks.

The two datasets released with this paper provide a foundation for further empirical investigations, contingent on awareness of their limitations and subjectivity. Continued advancements in both LLM architectures and collaborative workflows are needed to approach the goal of realistic, high-quality requirements-to-code datasets.