Using Large Language Models to Document Code: A First Quantitative and Qualitative Assessment

Abstract: Code documentation is vital for software development, improving readability and comprehension. However, it's often skipped due to its labor-intensive nature. AI LLMs present an opportunity to automate the generation of code documentation, easing the burden on developers. While recent studies have explored the use of such models for code documentation, most rely on quantitative metrics like BLEU to assess the quality of the generated comments. Yet, the applicability and accuracy of these metrics on this scenario remain uncertain. In this paper, we leveraged OpenAI GPT-3.5 to regenerate the Javadoc of 23,850 code snippets with methods and classes. We conducted both quantitative and qualitative assessments, employing BLEU alongside human evaluation, to assess the quality of the generated comments. Our key findings reveal that: (i) in our qualitative analyses, when the documents generated by GPT were compared with the original ones, 69.7% were considered equivalent (45.7%) or required minor changes to be equivalent (24.0%); (ii) indeed, 22.4% of the comments were rated as having superior quality than the original ones; (iii) the use of quantitative metrics is susceptible to inconsistencies, for example, comments perceived as having higher quality were unjustly penalized by the BLEU metric.