Integrating Code Metrics into Automated Documentation Generation for Computational Notebooks

Abstract: Effective code documentation is essential for collaboration, comprehension, and long-term software maintainability, yet developers often neglect it due to its repetitive nature. Automated documentation generation has evolved from heuristic and rule-based methods to neural network-based and LLM-based approaches. However, existing methods often overlook structural and quantitative characteristics of code that influence readability and comprehension. Prior research suggests that code metrics capture information relevant to program understanding. Building on these insights, this paper investigates the role of source code metrics as auxiliary signals for automated documentation generation, focusing on computational notebooks, a popular medium among data scientists that integrates code, narrative, and results but suffers from inconsistent documentation. We propose a two-stage approach. First, the CodeSearchNet dataset construction process was refined to create a specialized dataset from over 17 million code and markdown cells. After structural and semantic filtering, approximately 36,734 high-quality (code, markdown) pairs were extracted. Second, two modeling paradigms, a lightweight CNN-RNN architecture and a few-shot GPT-3.5 architecture, were evaluated with and without metric information. Results show that incorporating code metrics improves the accuracy and contextual relevance of generated documentation, yielding gains of 6% in BLEU-1 and 3% in ROUGE-L F1 for CNN-RNN-based architecture, and 9% in BERTScore F1 for LLM-based architecture. These findings demonstrate that integrating code metrics provides valuable structural context, enhancing automated documentation generation across diverse model families.

• The paper introduces a novel integration of code metrics into automated documentation generation, achieving improvements such as a 6% BLEU increase and a 9% BERTScore boost.
• It employs both CNN-RNN and GPT-3.5 architectures to evaluate how structural and quantitative metrics enhance documentation quality in computational notebooks.
• The study constructs a large, high-quality dataset and demonstrates that embedding code metrics significantly improves documentation coherence and contextual relevance.

Integrating Code Metrics into Automated Documentation Generation for Computational Notebooks

Introduction

The paper "Integrating Code Metrics into Automated Documentation Generation for Computational Notebooks" introduces a novel approach for enhancing code documentation in computational notebooks by incorporating structural and quantitative code metrics. This research addresses the inconsistency of documentation practices in computational notebooks, which combine narrative, code, and results but often suffer from documentation neglect.

Motivations and Objectives

Code metrics have been recognized for capturing relevant program information that aids in program comprehension. Despite advancements in documentation generation through neural networks and LLMs, most prior methods overlooked code metrics as auxiliary signals. This paper explores the integration of these metrics to improve documentation quality, focusing on computational notebooks commonly used by data scientists.

Methodology

The study introduces a refined dataset construction methodology, producing 36,734 high-quality code and markdown cell pairs from over 17 million initial candidates (Figure 1). Two distinct modeling paradigms are evaluated: a CNN-RNN architecture and a few-shot GPT-3.5 architecture. These architectures are analyzed with and without code metrics to assess their impact on documentation accuracy and contextual relevance.

Figure 1: The Dataset Construction Methodology.

Deep Learning-Based Architecture

The CNN-RNN architecture is chosen for its simplicity, enabling the assessment of code metrics' influence on models with constrained learning capacity. Augmenting the encoder input with code metrics resulted in a 6% increase in BLEU-1 and 3% in ROUGE-L F1 scores, attributed to better structural context in the generated documentation (Figure 2).

Figure 2: The CNN-RNN Architecture for Documentation Generation.

LLM-Based Architecture

In the LLM segment, the study adopts a few-shot learning approach with GPT-3.5, employing various shot samplers. Code metrics are integrated into the input representation through metric-informed prompting. The findings indicate that incorporating code metrics yields a remarkable 9% improvement in BERTScore F1 for the LLM-based model (Figure 3).

Figure 3: The Few-shot LLM-based Architecture for Documentation Generation.

Evaluation and Results

Evaluation metrics, including BLEU, ROUGE, and BERTScore, were employed to assess the quality of generated documentation (Figure 4). The proposed models outperform baseline approaches, demonstrating the benefit of integrating code metrics. These results reinforce the hypothesis that code metrics provide valuable structural context, enhancing the quality of automated documentation across different model families.

Figure 4: Overall Evaluation Results of the GPT-3.5 Model Across BLEU, ROUGE, and BERTScore Metrics.

Implications and Future Directions

The implications of this research are significant both theoretically and practically. Theoretically, it establishes a framework for leveraging code metrics in AI-driven documentation, linking quantitative measures with semantic understanding. Practically, the findings underscore the feasibility of improving documentation efficiency in computational environments. Future work could explore more sophisticated metric extraction methods or apply these insights across varied programming languages and platforms.

Conclusion

This study successfully demonstrates the integration of code metrics into automated documentation generation frameworks, yielding substantial improvements in documentation coherence and relevance for computational notebooks. These findings represent a meaningful step towards creating more intelligent, context-aware documentation tools, aligning with evolving software engineering practices.