Evaluating the Impact of ChatGPT on Exercises of a Software Security Course
Abstract: Along with the development of LLMs, e.g., ChatGPT, many existing approaches and tools for software security are changing. It is, therefore, essential to understand how security-aware these models are and how these models impact software security practices and education. In exercises of a software security course at our university, we ask students to identify and fix vulnerabilities we insert in a web application using state-of-the-art tools. After ChatGPT, especially the GPT-4 version of the model, we want to know how the students can possibly use ChatGPT to complete the exercise tasks. We input the vulnerable code to ChatGPT and measure its accuracy in vulnerability identification and fixing. In addition, we investigated whether ChatGPT can provide a proper source of information to support its outputs. Results show that ChatGPT can identify 20 of the 28 vulnerabilities we inserted in the web application in a white-box setting, reported three false positives, and found four extra vulnerabilities beyond the ones we inserted. ChatGPT makes nine satisfactory penetration testing and fixing recommendations for the ten vulnerabilities we want students to fix and can often point to related sources of information.
- OWASP, “Owasp top ten,” 2023. [Online]. Available: https://owasp.org/www-project-top-ten/
- Fireforx, “Firefox web developer tool.” [Online]. Available: https://firefox-dev.tools/
- Postman, “Postman,” 2023. [Online]. Available: https://www.postman.com/
- OWASP, “Owasp zed attack proxy.” [Online]. Available: https://owasp.org/www-project-zap/
- Tenable, “Nessus vulnerability assessment.” [Online]. Available: https://www.tenable.com/products/nessus
- OWASP, “Web security testing guide 4.2,” 2023. [Online]. Available: https://owasp.org/www-project-web-security-testing-guide/v42/
- OpenAI, “Chatgpt,” 2023. [Online]. Available: https://openai.com/blog/chatgpt
- ——, “Gpt4,” 2023. [Online]. Available: https://openai.com/product/gpt-4
- B. A. Becker, P. Denny, J. Finnie-Ansley, A. Luxton-Reilly, J. Prather, and E. A. Santos, “Programming is hard–or at least it used to be: Educational opportunities and challenges of ai code generation,” arXiv preprint arXiv:2212.01020, 2022.
- J. Finnie-Ansley, P. Denny, B. A. Becker, A. Luxton-Reilly, and J. Prather, “The robots are coming: Exploring the implications of openai codex on introductory programming,” in Proceedings of the 24th Australasian Computing Education Conference, ser. ACE ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 10–19. [Online]. Available: https://doi.org/10.1145/3511861.3511863
- C. Tony, M. Balasubramanian, N. E. Díaz Ferreyra, and R. Scandariato, “Conversational devbots for secure programming: An empirical study on skf chatbot,” in Proceedings of the International Conference on Evaluation and Assessment in Software Engineering 2022, 2022, pp. 276–281.
- M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021.
- F. Fischer, K. Böttinger, H. Xiao, C. Stransky, Y. Acar, M. Backes, and S. Fahl, “Stack overflow considered harmful? the impact of copy&paste on android application security,” in 2017 IEEE Symposium on Security and Privacy (SP). IEEE, 2017, pp. 121–136.
- N. Perry, M. Srivastava, D. Kumar, and D. Boneh, “Do users write more insecure code with ai assistants?” arXiv preprint arXiv:2211.03622, 2022.
- S. Santhanam, T. Hecking, A. Schreiber, and S. Wagner, “Bots in software engineering: a systematic mapping study,” PeerJ Computer Science, vol. 8, p. e866, 2022.
- Secure Code Warrior, “Secure code warrior.” [Online]. Available: https://www.securecodewarrior.com/
- OpenAI, “Gpt-4 technical report,” 2023.
- M. Meucci and A. Muller, “Owasp testing guide, v4,” OWASP Foundation, vol. 4, pp. 14–23, 2014.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.