On Using Stack Overflow Comment-Edit Pairs to recommend code maintenance changes

Abstract: Code maintenance data sets typically consist of a before and after version of the code that contains the improvement or fix. Such data sets are important for software engineering support tools related to code maintenance, such as program repair, code recommender systems, or Application Programming Interface (API) misuse detection. Most of the current data sets are constructed from mining commit history in version-control systems or issues in issue-tracking systems. In this paper, we investigate whether Stack Overflow can be used as an additional data source. Comments on Stack Overflow provide an effective way for developers to point out problems with existing answers, alternative solutions, or pitfalls. In this paper, we mine comment-edit pairs from Stack Overflow and investigate their potential usefulness. These pairs have the added benefit of having concrete descriptions of why the change is needed as well as potentially having less tangled changes to deal with. We first design a technique to extract related comment-edit pairs and then investigate the nature of these pairs. We find that the majority of comment-edit pairs are not tangled, but only 27% of the studied pairs are potentially useful for the above applications. We categorize the types of mined pairs and find that the highest ratio of useful pairs come from categories Correction, Obsolete, Flaw, and Extension. To demonstrate the effectiveness of our extracted pairs, we submitted 15 pull requests on GitHub, 10 of which have been accepted to widely used repositories such as Apache Beam and nltk. Our work is the first to investigate Stack Overflow comment-edit pairs and opens the door for future work in this direction. Based on our findings and observations, we provide concrete suggestions on how to potentially identify a larger set of useful comment-edit pairs, which can also be facilitated by our shared data.