Advancing GenAI Assisted Programming--A Comparative Study on Prompt Efficiency and Code Quality Between GPT-4 and GLM-4
Abstract: This study aims to explore the best practices for utilizing GenAI as a programming tool, through a comparative analysis between GPT-4 and GLM-4. By evaluating prompting strategies at different levels of complexity, we identify that simplest and straightforward prompting strategy yields best code generation results. Additionally, adding a CoT-like preliminary confirmation step would further increase the success rate. Our results reveal that while GPT-4 marginally outperforms GLM-4, the difference is minimal for average users. In our simplified evaluation model, we see a remarkable 30 to 100-fold increase in code generation efficiency over traditional coding norms. Our GenAI Coding Workshop highlights the effectiveness and accessibility of the prompting methodology developed in this study. We observe that GenAI-assisted coding would trigger a paradigm shift in programming landscape, which necessitates developers to take on new roles revolving around supervising and guiding GenAI, and to focus more on setting high-level objectives and engaging more towards innovation.
- Methods and applications of ChatGPT in software development: A literature review, Southeast Europe Journal of Soft Computing 12 (2023) 08–12.
- J. Hendler, Understanding the limits of AI coding, Science 379 (2023) 548–548. URL: https://www.science.org/doi/abs/10.1126/science.adg4246. doi:10.1126/science.adg4246. arXiv:https://www.science.org/doi/pdf/10.1126/science.adg4246.
- Codereval: A benchmark of pragmatic code generation with generative pre-trained models, in: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–12.
- Better together? an evaluation of AI-supported code translation, in: 27th International Conference on Intelligent User Interfaces, IUI ’22, ACM, 2022. URL: http://dx.doi.org/10.1145/3490099.3511157. doi:10.1145/3490099.3511157.
- M. Jonsson, J. Tholander, Cracking the code: Co-coding with AI in creative programming education, in: Proceedings of the 14th Conference on Creativity and Cognition, Association for Computing Machinery, New York, NY, USA, 2022, p. 5–14. URL: https://doi.org/10.1145/3527927.3532801. doi:10.1145/3527927.3532801.
- Snake game AI: Movement rating functions and evolutionary algorithm-based optimization, in: 2016 Conference on Technologies and Applications of Artificial Intelligence (TAAI), 2016, pp. 256–261. doi:10.1109/TAAI.2016.7880166.
- A business classifier to detect readability metrics on software games and their types, International Journal of E-Entrepreneurship and Innovation (IJEEI) 4 (2013) 47–57.
- A. Theodoraki, S. Xinogalos, Studying students’ attitudes on using examples of game source code for learning programming, Informatics in Education 13 (2014) 265–277.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.