Evaluating Large Language Models on Computer Science University Exams in Data Structures

Abstract: We present a comprehensive evaluation of LLMs on Computer Science (CS) Data Structure examination questions. Our work introduces a new benchmark dataset comprising exam questions from Tel Aviv University (TAU), curated to assess LLMs' abilities in handling closed and multiple-choice questions. We evaluated the performance of OpenAI's GPT 4o and Anthropic's Claude 3.5, popular LLMs, alongside two smaller LLMs, Mathstral 7B and LLaMA 3 8B, across the TAU exams benchmark. Our findings provide insight into the current capabilities of LLMs in CS education.

• The paper introduces a novel benchmark derived from 57 university exams, with questions translated to LaTeX and manually validated.
• The paper reports that high-capacity models (GPT-4o and Claude 3.5) achieved 56-61% accuracy on multiple-choice questions, outperforming smaller models.
• The paper highlights that chain-of-thought prompting yielded minimal gains, recommending open-ended assessments to safeguard academic integrity.

Evaluation of LLMs on University Data Structures Exams: A Technical Review

Benchmark Construction and Methodology

The study introduces a novel benchmark dataset derived from 57 university-level Data Structures exams at Tel Aviv University, encompassing questions from 2001–2018. This benchmark targets theoretical content, emphasizing closed and multiple-choice (types B, C, D) questions. Questions were translated into latex-formatted English and manually validated to ensure fidelity.

Four LLMs were evaluated: OpenAI GPT-4o, Anthropic Claude 3.5, Mathstral 7B, and LLaMA 3 8B. Each question was presented to each model five times under two prompting regimes: standard and chain-of-thought (CoT). A two-stage querying protocol was adopted, requiring the model to reason before outputting a final answer. Performance was assessed using strict matching for multiple-choice questions and manual evaluation (50% sample) for closed-ended questions to account for lexical variability.

Figure 1: Distribution of possible answers for each multiple-choice question, providing a statistical lower bound for random guessing accuracy.

Comparative Performance Analysis

GPT-4o and Claude 3.5 exhibited substantially higher performance than random guessing (average random guess probability: 0.28), achieving 56.71% (GPT-4o) and 61.53% (Claude 3.5) accuracy on multiple-choice questions (types C/D), and 46.67% (GPT-4o) and 49.33% (Claude 3.5) on closed-ended questions (type B). Smaller models Mathstral and LLaMA 3 demonstrated markedly lower accuracy (<32% across all question types), indicating that parameter scale and training corpus breadth are crucial for theoretical reasoning tasks.

Majority voting (≥3 correct out of 5) provided a slightly higher success rate, reflecting modest model consistency. Notably, Claude 3.5 was more “decisive”—producing identical answers across repetitions more frequently—compared to GPT-4o, though temperature settings may influence this effect.

Figure 2: Model accuracy stratified by question type and success criterion, highlighting the performance differential against random guessing.

Figure 3: Distribution of correct repetitions (out of 5) with CoT, illustrating inter-model consistency in response accuracy.

Topic-Level Breakdown and Prompt Engineering

Topic-stratified analysis revealed that both large LLMs achieved higher success rates on questions involving arrays, linked lists, hash tables, complexity analysis, and sorting lower bounds. Performance decreased considerably on topics demanding nuanced mathematical reasoning, such as amortized analysis and selection algorithms.

Empirically, CoT prompting failed to deliver meaningful accuracy gains across models and question types. This result is attributed to the possibility that CoT-style reasoning is already implicitly embedded in the models’ inference pipelines. Therefore, prompt engineering yields diminishing returns for high-capacity models in the context of university-theory questions.

Implications and Recommendations

Results substantiate that contemporary LLMs (GPT-4o, Claude 3.5) can competently, but not reliably, answer university-level theoretical CS questions. Their strengths are most pronounced in canonical, well-represented topics, likely due to widespread coverage in pretraining corpora. However, their limitations become apparent in domains requiring complex, multi-step reasoning or creative analysis.

From an educational perspective, reliance on LLMs for assessment risks undermining academic integrity for question types amenable to model solution. Therefore, assessments should favor open-ended and abstract questions where LLMs are less effective. Additionally, supervised, in-person testing is advocated to ensure that students' independent competencies are accurately appraised.

Practically, LLMs offer utility as learning aids for practice and concept clarification, but their unreliability precludes their use as sole evaluative instruments. Theoretical development in AI should prioritize enhancing models’ reasoning over mathematical and abstract content, possibly via specialized finetuning or augmentation with symbolic solvers. Furthermore, future research should dissect error modes and explore architectural and training innovations to bridge this reliability gap.

Conclusion

This evaluation demonstrates that state-of-the-art LLMs deliver moderately high accuracy on university-level data structures exams, especially for constrained answer formats and fundamental topics. However, their limitations in covering abstract or mathematically rigorous questions highlight the necessity for caution in integrating LLMs into educational assessment. Continued advancement in model design and benchmark complexity is essential for furthering AI’s pedagogical and evaluative reliability.