CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs

Abstract: Recent advances in Code LLMs (CodeLLMs) have primarily focused on open-ended code generation, often overlooking the crucial aspect of code understanding and reasoning. To bridge this gap, we introduce CodeMMLU, a comprehensive multiple-choice benchmark designed to evaluate the depth of software and code comprehension in LLMs. CodeMMLU includes nearly 20,000 questions spanning diverse domains, including code analysis, defect detection, and software engineering principles across multiple programming languages. Unlike traditional benchmarks that emphasize code generation, CodeMMLU assesses a model's ability to reason about programs across a wide-range of tasks such as code repair, execution reasoning, and fill-in-the-blank challenges. Our extensive evaluation reveals that even state-of-the-art models struggle with CodeMMLU, highlighting significant gaps in comprehension beyond generation. By emphasizing the essential connection between code understanding and effective AI-assisted development, CodeMMLU provides a critical resource for advancing more reliable and capable coding assistants.

• The paper introduces CodeMMLU, a benchmark featuring over 10,000 multiple-choice tasks to evaluate code comprehension across various software engineering domains.
• The paper reveals that even advanced models like GPT-4 struggle with complex code understanding, highlighting significant limitations in current CodeLLMs.
• The benchmark underscores that traditional scaling laws and advanced prompting techniques do not consistently enhance code comprehension performance.

Overview of CodeMMLU: A Benchmark for Code Understanding in LLMs

The paper presents CodeMMLU, a benchmark designed to evaluate the comprehension abilities of Code LLMs (CodeLLMs). Unlike traditional benchmarks, CodeMMLU emphasizes code understanding rather than generation, addressing a critical gap in evaluating software-related knowledge.

Key Contributions

1. Benchmark Design:

CodeMMLU is structured as a multiple-choice question-answer (MCQA) benchmark comprising over 10,000 questions that span various software engineering domains and programming languages. The benchmark includes tasks in code analysis, defect detection, and comprehension of software principles.

2. Evaluation and Findings:

The evaluation of state-of-the-art models indicates significant challenges in understanding complex software concepts. The benchmark's results highlight deficiencies in code comprehension, revealing the limitations of existing models beyond mere code generation.

3. Insights into CodeLLMs:

Several key findings emerge from the analysis:

• GPT-4 leads in accuracy among closed-source models, while Meta-Llama-3 stands out in open-source offerings.
• Traditional scaling laws regarding model size and performance are not consistently observed.
• Advanced prompting techniques such as Chain-of-Thought (CoT) do not always enhance performance.

Implications

Practical Implications:

The CodeMMLU benchmark is instrumental in advancing AI-assisted software development by facilitating the creation of more reliable coding assistants. It underscores the need for balanced model capabilities that integrate both generation and comprehension.

Theoretical Implications:

The insights from CodeMMLU contribute to understanding the intricate relationship between model architecture, training data quality, and performance in software domains. It challenges researchers to develop methodologies that address these complexities.

Future Directions

CodeMMLU sets a foundation for future research aimed at refining model evaluation techniques. The benchmark suggests paths for developing models that can more effectively comprehend and reason about code, potentially revolutionizing AI's role in software engineering.

Conclusion

By introducing CodeMMLU, the paper provides the research community with a comprehensive tool to assess and improve the understanding capabilities of CodeLLMs. This contribution is vital in the ongoing effort to enhance the reliability and effectiveness of AI in software development tasks.