Assessing the Software Security Comprehension of Large Language Models

Abstract: LLMs are increasingly used in software development, but their level of software security expertise remains unclear. This work systematically evaluates the security comprehension of five leading LLMs: GPT-4o-Mini, GPT-5-Mini, Gemini-2.5-Flash, Llama-3.1, and Qwen-2.5, using Blooms Taxonomy as a framework. We assess six cognitive dimensions: remembering, understanding, applying, analyzing, evaluating, and creating. Our methodology integrates diverse datasets, including curated multiple-choice questions, vulnerable code snippets (SALLM), course assessments from an Introduction to Software Security course, real-world case studies (XBOW), and project-based creation tasks from a Secure Software Engineering course. Results show that while LLMs perform well on lower-level cognitive tasks such as recalling facts and identifying known vulnerabilities, their performance degrades significantly on higher-order tasks that require reasoning, architectural evaluation, and secure system creation. Beyond reporting aggregate accuracy, we introduce a software security knowledge boundary that identifies the highest cognitive level at which a model consistently maintains reliable performance. In addition, we identify 51 recurring misconception patterns exhibited by LLMs across Blooms levels.

• The paper introduces the BASKET framework, applying Bloom’s Taxonomy to assess LLMs’ software security skills.
• Empirical results reveal strong factual recall yet notable performance drops in complex reasoning tasks and creative security synthesis.
• The study identifies 51 recurring security misconceptions and formalizes a knowledge boundary metric to gauge LLM reliability.

Assessing the Software Security Comprehension of LLMs

Introduction and Motivation

The adoption of LLMs in software engineering workflows for tasks such as code generation, vulnerability detection, and informal tutoring has accelerated, yet their true comprehension of software security remains underexplored. "Assessing the Software Security Comprehension of LLMs" (2512.21238) systematically evaluates the security competence of leading LLMs—GPT-4o-Mini, GPT-5-Mini, Gemini-2.5-Flash, Llama-3.1, and Qwen-2.5—through a cognitively grounded lens. The work introduces BASKET, a Bloom’s Taxonomy-guided framework, establishing a multi-level diagnostic for software security expertise by probing six cognitive domains: remembering, understanding, applying, analyzing, evaluating, and creating. The empirical methodology leverages diverse datasets, including educational MCQs, curated code vulnerability corpora (SALLM), project-based tasks, and case study analysis. This structured approach yields both quantitative measures of knowledge boundaries and a taxonomy of systematic misconceptions, illuminating how LLMs transfer, degrade, or fail in security contexts.

Figure 1: Overview of BASKET—a Bloom’s Taxonomy-Guided Framework for Evaluating Software Security Knowledge.

Methodology: The BASKET Framework

The evaluation pipeline consists of (1) selection of a software security curriculum anchored in the CyBOK Software Security Knowledge Area, and (2) systematic task curation mapped to Bloom’s six cognitive levels. Each level probes a distinct cognitive function, from rote factual recall to the design of secure systems. Assessment instruments are sourced from a blend of paraphrased online and MOOC-based MCQs (to mitigate contamination), course quizzes and projects, the SALLM code vulnerability dataset (45 CWEs, 100 Python code prompts), and case studies from XBOW’s project-scale benchmarks.

Explicit and implicit prompting strategies are deployed to differentiate between surface-level and deeper security comprehension. The evaluation rubric—iteratively developed and cross-validated among experts—emphasizes conceptual rigor, reasoning depth, and solution specificity rather than binary correctness.

The prompting templates for varying SQs (subquestions aligning with Bloom’s levels) are meticulously designed to control for instruction-following and prompt sensitivity (Figure 2, Figure 3, Figure 4, Figure 5).

Figure 2: Standardized prompt used for LLM MCQ answering in the “remember” (SQ1) evaluation.

Figure 3: Prompt template used in the “understanding” assessment for evaluating fundamental software security knowledge (SQ2).

Figure 4: Prompt adopted for the “evaluating” case study (SQ5) task leveraging XBOW benchmarks.

Figure 5: Project-based prompt template for probing “creating” level security synthesis (SQ6).

Empirical Results Across Cognitive Levels

Lower-Order Cognitive Tasks

At the "remember" and "understand" levels, LLMs demonstrate strong factual recall and basic security concept identification. Top models exceed 0.84 pass@1 on internet/MCQ benchmarks and maintain mean scores above 4.0/5.0 on short-answer quizzes, with closed-source models (GPT-4o-Mini, Gemini-2.5-Flash) outperforming open models. Standard deviation analysis confirms high answer stability for GPT-based models.

Mid-Level Tasks: Application and Analysis

Applying and analyzing knowledge exposes significant performance stratification. On scenario-driven SALLM tasks, GPT-5-Mini and Gemini-2.5-Flash maintain near-perfect mean scores in applied exploit identification and code repair, especially with explicit vulnerability prompts. Llama-3.1 and Qwen-2.5 exhibit degraded reliability and inconsistency as task complexity increases, particularly under higher sampling temperatures.

Higher-Order Tasks: Evaluation and Creation

The performance gap widens on evaluation and creation. In threat modeling case studies (XBOW), GPT-5-Mini and Gemini-2.5-Flash consistently generate complete and domain-appropriate reports, meeting or exceeding rubrical thresholds for architectural overview, asset identification, threat documentation, and risk assessment. Llama-3.1 and Qwen-2.5 frequently fail to meet minimum criteria, and their outputs lack structural and analytical sufficiency.

Project-level creation tasks reinforce this pattern—while Gemini-2.5-Flash or GPT models may excel in isolated project scenarios, no model consistently achieves high rubric scores across all creation tasks. Open-source models are especially vulnerable to task ambiguity and synthesis failure, often defaulting to superficial or generic advice.

Knowledge Boundary Formalization

A formal "knowledge boundary" metric—$K_B(m)$—is introduced, defined as the highest Bloom level for which the model reliably exceeds a normalized performance threshold $\tau$ across all preceding levels. Most models sustain competence only up to mid-Bloom levels (Apply/Analyze) for $\tau = 0.7$ (70% proficiency). Remarkably, only GPT-5-Mini demonstrates robust, threshold-insensitive boundaries at the "Create" level. Minor threshold increments sharply contract the operational boundaries of Llama-3.1 and Qwen-2.5, highlighting their fragility (Figure 6).

Figure 6: Overall evaluation of models at low/high temperature—LLMs are robust at lower-order tasks but generally unreliable for higher-order reasoning and creation.

Software Security Misconception Taxonomy

A novel taxonomy of 51 recurring software security misconception patterns is extracted via open coding of substandard and incorrect LLM outputs. These misconceptions range from erroneous attack class invention and confusion of core security principles to architectural reasoning failures at the system design level. Patterns are observed to be systematic, domain-specific, and persistent across both open- and closed-source models (Figure 7).

Figure 7: Taxonomy of LLM software security misconceptions spanning all six Bloom cognitive levels.

Notably, several LLM misconceptions mirror those in security education literature, such as conflation of authentication/authorization, overgeneralization of defense mechanisms (e.g., ORMs preventing all SQL injection), and overconfidence in tool-based mitigation. However, LLMs also introduce errors atypical of human learners, including the hallucination of non-standard vulnerability classes and unwarranted trust in superficial code changes.

Implications for Research, Practice, and Future Model Development

The findings have important implications:

• Benchmarking: Task-based accuracy is insufficient—LLMs must be assessed across cognitive levels, with focus on qualitative, not just quantitative, failure patterns.
• Model Design: Enhancing LLM reliability on higher-order security reasoning requires advances in contextual grounding, composition, and architectural understanding.
• Practical Deployment: LLMs can be trusted for rote recall and identification support but are currently unreliable for autonomous security evaluation or system-level design without expert oversight.
• Educational Use: The taxonomy of misconceptions enables educators to design interventions targeting LLM-propagated errors and can inform the development of automated feedback scaffolding.
• Theoretical Insights: The concept of the knowledge boundary, previously defined in factual recall settings, is shown to require a cognitively nuanced, domain-specific interpretation when applied to complex reasoning in software security.
• Future Directions: Integration of uncertainty estimation, abstention strategies, and hybrid symbolic-LLM reasoning may help mitigate knowledge boundary failures in future models.

Conclusion

The paper provides a rigorously structured, multi-faceted assessment revealing that current LLMs, while effective at factual recall and basic security identification, degrade sharply in reliability as tasks demand reasoning, judgment, or creative secure system synthesis. The systematic extraction of misconception patterns demonstrates that these failures are not isolated but often patterned and recurring. The introduction of the knowledge boundary construct, mapped to Bloom’s taxonomy, frames a practical constraint for LLM deployment in security-critical workflows and underscores the need for further research into model calibration, reasoning depth, and educative alignment for trustworthy AI systems in software security domains.