Papers
Topics
Authors
Recent
Search
2000 character limit reached

Predicting Line-Level Defects by Capturing Code Contexts with Hierarchical Transformers

Published 19 Dec 2023 in cs.SE and cs.AI | (2312.11889v1)

Abstract: Software defects consume 40% of the total budget in software development and cost the global economy billions of dollars every year. Unfortunately, despite the use of many software quality assurance (SQA) practices in software development (e.g., code review, continuous integration), defects may still exist in the official release of a software product. Therefore, prioritizing SQA efforts for the vulnerable areas of the codebase is essential to ensure the high quality of a software release. Predicting software defects at the line level could help prioritize the SQA effort but is a highly challenging task given that only ~3% of lines of a codebase could be defective. Existing works on line-level defect prediction often fall short and cannot fully leverage the line-level defect information. In this paper, we propose Bugsplorer, a novel deep-learning technique for line-level defect prediction. It leverages a hierarchical structure of transformer models to represent two types of code elements: code tokens and code lines. Unlike the existing techniques that are optimized for file-level defect prediction, Bugsplorer is optimized for a line-level defect prediction objective. Our evaluation with five performance metrics shows that Bugsplorer has a promising capability of predicting defective lines with 26-72% better accuracy than that of the state-of-the-art technique. It can rank the first 20% defective lines within the top 1-3% suspicious lines. Thus, Bugsplorer has the potential to significantly reduce SQA costs by ranking defective lines higher.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. “IEEE Standard Glossary of Software Engineering Terminology” In IEEE Std 610.12-1990, 1990, pp. 1–84
  2. R.L. Glass “Frequently forgotten fundamental facts about software engineering” In IEEE Software 18.3, 2001, pp. 112–111
  3. “Reversible Debugging Software: Quantify the time and cost saved using reversible debuggers” In University Cambridge: Cambridge, UK, 2013
  4. “How practitioners perceive automated bug report management techniques” In TSE 46.8 IEEE, 2018, pp. 836–862
  5. “Investigating code review practices in defective files: An empirical study of the qt system” In 2015 IEEE/ACM 12th WCMSR, 2015, pp. 168–179 IEEE
  6. “Revisiting code ownership and its relationship with software quality in the scope of modern code review” In Proceedings of the 38th ICSE, 2016, pp. 1039–1050
  7. “DeepLineDP: Towards a deep learning approach for line-level defect prediction” In IEEE TSE IEEE, 2022
  8. “Revisiting the impact of dependency network metrics on software defect prediction” In IEEE TSE 48.12 IEEE, 2021, pp. 5030–5049
  9. “An empirical study of learning to rank techniques for effort-aware defect prediction” In 2019 IEEE 26th SANER, 2019, pp. 298–309 IEEE
  10. Jirayus Jiarpakdee, Chakkrit Kla Tantithamthavorn and John Grundy “Practitioners’ perceptions of the goals and visual explanations of defect prediction models” In 2021 IEEE/ACM 18th MSR, 2021, pp. 432–443 IEEE
  11. “Software visualization and deep transfer learning for effective software defect prediction” In Proceedings of the ACM/IEEE 42nd ICSE, 2020, pp. 578–589
  12. Thomas Shippey, David Bowes and Tracy Hall “Automatically identifying code features for software defect prediction: Using AST N-grams” In Information and Software Technology 106 Elsevier, 2019, pp. 142–160
  13. “Predicting defective lines using a model-agnostic technique” In IEEE, TSE IEEE, 2020
  14. Chanathip Pornprasit and Chakkrit Kla Tantithamthavorn “Jitline: A simpler, better, faster, finer-grained just-in-time defect prediction” In 2021 IEEE/ACM 18th MSR, 2021, pp. 369–379 IEEE
  15. “Attention is all you need” In Advances in NeurIPS 30, 2017
  16. “Sentence-bert: Sentence embeddings using siamese bert-networks” In arXiv preprint arXiv:1908.10084, 2019
  17. Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio “Neural machine translation by jointly learning to align and translate” In arXiv preprint arXiv:1409.0473, 2014
  18. Parvez Mahbub, Ohiduzzaman Shuvo and M.Masudur Rahman “Defectors: A Large, Diverse Python Dataset for Defect Prediction” In Proceeding of The 20th MSR, 2023, pp. 5
  19. “CodeBERT: A Pre-Trained Model for Programming and Natural Languages” In Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 1536–1547
  20. “CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation” In Proceedings of the 2021 EMNLP, 2021, pp. 8696–8708
  21. Rico Sennrich, Barry Haddow and Alexandra Birch “Neural Machine Translation of Rare Words with Subword Units” In Proceedings of the 54th ACL, 2016, pp. 1715–1725
  22. “Bert: Pre-training of deep bidirectional transformers for language understanding” In arXiv preprint arXiv:1810.04805, 2018
  23. “Roberta: A robustly optimized bert pretraining approach” In arXiv preprint arXiv:1907.11692, 2019
  24. Cynthia Dwork “The mathematics of information coding, extraction, and distribution” In The IMA Volumes in Mathematics and its applications 107 Springer, 1999, pp. 82
  25. “Decoupled weight decay regularization” In arXiv preprint arXiv:1711.05101, 2017
  26. Diederik P Kingma and Jimmy Ba “Adam: A method for stochastic optimization” In arXiv preprint arXiv:1412.6980, 2014
  27. Jacek Śliwerski, Thomas Zimmermann and Andreas Zeller “When do changes induce fixes?” In ACM sigsoft software engineering notes 30.4 ACM New York, NY, USA, 2005, pp. 1–5
  28. “Recommendations for Datasets for Source Code Summarization” In Proceedings of NAACL-HLT, 2019, pp. 3931–3937
  29. “DeepJIT: an end-to-end deep learning framework for just-in-time defect prediction” In 2019 IEEE/ACM 16th MSR, 2019, pp. 34–45 IEEE
  30. Ryan J Urbanowicz and Jason H Moore “ExSTraCS 2.0: description and evaluation of a scalable learning classifier system” In Evolutionary intelligence 8 Springer, 2015, pp. 89–116
  31. César Ferri, José Hernández-Orallo and R Modroiu “An experimental comparison of performance measures for classification” In Pattern recognition letters 30.1 Elsevier, 2009, pp. 27–38
  32. “Codesearchnet challenge: Evaluating the state of semantic code search” In arXiv preprint arXiv:1909.09436, 2019
  33. “Language models are unsupervised multitask learners” In OpenAI blog 1.8, 2019, pp. 9
  34. Parvez Mahbub, Ohiduzzaman Shuvo and Mohammad Masudur Rahman “Explaining Software Bugs Leveraging Code Structures in Neural Machine Translation” In 2023 IEEE/ACM 45th ICSE, 2023 IEEE
  35. “On the Evaluation of Commit Message Generation Models: An Experimental Study” In 2021 ICSME, 2021, pp. 126–136 IEEE
  36. “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer” In JMLR 21, 2020, pp. 1–67
  37. “Codexglue: A machine learning benchmark dataset for code understanding and generation” In arXiv preprint arXiv:2102.04664, 2021
  38. Ohiduzzaman Shuvo, Parvez Mahbub and Mohammad Masudur Rahman “Recommending Code Reviews Leveraging Code Changes with Structured Information Retrieval” In 2023 29th IEEE ICSME, 2023 IEEE
  39. Saikat Mondal, Mohammad Masudur Rahman and Chanchal K Roy “Can issues reported at stack overflow questions be reproduced? an exploratory study” In 2019 IEEE/ACM 16th MSR, 2019, pp. 479–489 IEEE
  40. Gema Rodriguez-Perez, Meiyappan Nagappan and Gregorio Robles “Watch out for extrinsic bugs! a case study of their impact in just-in-time bug prediction models on the openstack project” In IEEE TSE 48.4 IEEE, 2020, pp. 1400–1416
  41. “Revisiting common bug prediction findings using effort-aware models” In 2010 IEEE ICSME, 2010, pp. 1–10 IEEE
  42. Hideaki Hata, Osamu Mizuno and Tohru Kikuno “Bug prediction based on fine-grained module histories” In 2012 34th ICSE, 2012, pp. 200–210 IEEE
  43. “Are Fix-Inducing Changes a Moving Target? A Longitudinal Case Study of Just-In-Time Defect Prediction” In IEEE TSE 44.5, 2018, pp. 412–428 DOI: 10.1109/TSE.2017.2693980
  44. Luca Pascarella, Fabio Palomba and Alberto Bacchelli “Fine-grained just-in-time defect prediction” In Journal of Systems and Software 150 Elsevier, 2019, pp. 22–36
  45. Qiao Huang, Xin Xia and David Lo “Revisiting supervised and unsupervised models for effort-aware just-in-time defect prediction” In Empirical Software Engineering 24.5 Springer, 2019, pp. 2823–2862
  46. “Cc2vec: Distributed representations of code changes” In Proceedings of the ACM/IEEE 42nd ICSE, 2020, pp. 518–529
  47. “A large-scale empirical study of just-in-time quality assurance” In IEEE TSE 39.6 IEEE, 2012, pp. 757–773
  48. “Perceptions, expectations, and challenges in defect prediction” In IEEE TSE 46.11 IEEE, 2018, pp. 1241–1266
  49. Tian Jiang, Lin Tan and Sunghun Kim “Personalized defect prediction” In 2013 28th IEEE/ACM ASE, 2013, pp. 279–289 Ieee
  50. Song Wang, Taiyue Liu and Lin Tan “Automatically learning semantic features for defect prediction” In Proceedings of the 38th International Conference on Software Engineering, 2016, pp. 297–308
  51. “Software defect prediction via convolutional neural network” In 2017 IEEE international conference on software quality, reliability and security (QRS), 2017, pp. 318–328 IEEE
  52. “Improving bug detection via context-based code representation learning and attention-based neural networks” In Proceedings of the ACM on Programming Languages 3.OOPSLA ACM New York, NY, USA, 2019, pp. 1–30
  53. “Automatic feature learning for predicting vulnerable software components” In IEEE TSE 47.1 IEEE, 2018, pp. 67–85
  54. “VulDeePecker: A Deep Learning-Based System for Multiclass Vulnerability Detection” In IEEE Transactions on Dependable and Secure Computing 18.5 IEEE, 2019, pp. 2224–2236
  55. “To what extent could we detect field defects? An extended empirical study of false negatives in static bug-finding tools” In ASE 22.4 Springer, 2015, pp. 561–602
  56. Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin ““Why should I trust you?” Explaining the predictions of any classifier” In Proceedings of the 22nd ACM SIGKDD KDDM, 2016, pp. 1135–1144
  57. “Learning phrase representations using RNN encoder-decoder for statistical machine translation” In arXiv preprint arXiv:1406.1078, 2014

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free