CWEval: Benchmarking Secure Code Performance

• CWEval is a comprehensive framework that integrates dynamic execution and static analysis to measure code security and functional accuracy against industry-standard CWE criteria.
• It employs outcome-driven test oracles and dual evaluation metrics to assess LLM-generated code, highlighting gaps between functional correctness and security compliance.
• CWEval’s curated benchmarks and datasets, such as CWEval-bench and CASTLE, enable reproducible comparisons and drive improvements in automated vulnerability mapping.

CWEval is a term that has come to encompass a set of methodologies, benchmarks, datasets, and evaluation protocols for systematically assessing the security and functional correctness of software artifacts—especially code produced by LLMs—with respect to community-standard Common Weakness Enumeration (CWE) vulnerability classes. Its instantiations span outcome-driven functional-security benchmarks, automated CVE-to-CWE mapping via machine learning, curated datasets for tool validation, and empirical measurement frameworks for quantifying and comparing the vulnerability surface of codebases. While the name CWEval is sometimes used for particular benchmarks (e.g., CWEval-bench (Peng et al., 14 Jan 2025), or the manually verified suite in (&&&1&&&)), it also refers generically to any rigorous evaluation pipeline that maps code or vulnerability disclosures to the CWE taxonomy and reports results with reproducible, statistically grounded metrics.

1. Origins and Core Concepts of CWEval

The development of CWEval is motivated by critical deficiencies in prior security evaluation methodologies for code and vulnerabilities. Existing benchmarks such as CyberSecEval and SecurityEval either lacked precise task specifications, suffered from static analysis instability, or decoupled security verdicts from the actual functional intent of the code. CWEval addresses these gaps through the introduction of:

• Outcome-driven, dynamic oracles: Rather than relying solely on static pattern matching, CWEval’s test oracles execute candidate solutions using controlled inputs. This enables detection of subtler vulnerability manifestations, reducing both false positives and negatives (Peng et al., 14 Jan 2025).
• Joint evaluation of security and functionality: For each task, CWEval computes both functional correctness (does the code operate as specified?) and security compliance (does it avoid the targeted CWE?) simultaneously. This is central to the CWEval-bench and Secure-Instruct evaluation protocols (Li et al., 8 Oct 2025).
• Alignment with the CWE taxonomy: Automated or evaluative mappings always correspond to official MITRE CWE definitions, supporting granular measurement and standardized reporting across studies (Shahid et al., 24 Nov 2025, Aghaei et al., 2020).

2. Dataset Construction and Task Scope

CWEval for LLM Code Generation (CWEval-bench, (Peng et al., 14 Jan 2025, Li et al., 8 Oct 2025))

CWEval-bench consists of 119 security-critical programming tasks spanning 31 distinct CWEs. Each scenario includes:

• A precise natural-language specification (no explicit security hints).
• Two suites of executable test oracles:
  • Functional tests: validate specification compliance.
  • Security tests: assert the absence of a specific vulnerability (CWE).
• Self-contained code samples in five languages (Python, Java, JavaScript, Go, C).
• Reference implementations exemplifying both vulnerable (functionally correct but insecure) and secure (functionally and security correct) solutions.

All scenarios and tests are manually reviewed for correctness and specificity. The resulting dataset enables direct measurement of both secure code generation and functional regression from security-focused interventions.

CWE-based Static Analysis and CVE Mapping (Shahid et al., 24 Nov 2025, Haddad et al., 2023, Aghaei et al., 2020)

Alternative uses of "CWEval" refer to:

• Synthetic prompt-based LLM evaluation: 84 hand-written prompts (one per CWE) to elicit targeted vulnerabilities for empirical analysis of LLM-generated code.
• Manually annotated CVE-to-CWE datasets: For semantic mapping of real-world vulnerability disclosures (CVE records) to the CWE hierarchy, supporting tool training and evaluation.
• CASTLE microbenchmark suite (Dubniczky et al., 12 Mar 2025): 250 C programs, each annotated at the line level for a single CWE instance; supports static analysis and formal verification tools alongside LLM-based methods.

3. Evaluation Protocols and Metrics

Code Generation Benchmarks (Peng et al., 14 Jan 2025, Li et al., 8 Oct 2025)

For each benchmark scenario, the evaluation protocol is as follows:

• Generation: LLMs produce n = 100 samples per scenario at a specified temperature (typically 0.8).
• Metrics: Top-1 pass rates are computed for:
  • Func@1: fraction of scenarios whose top-1 solution passes all functional tests.
  • Sec@1: passes all security tests.
  • Func-Sec@1: passes both test suites simultaneously.

More generally, unbiased pass@k is defined by: $$Metric_{\theta}@k = 1 - \frac{\sum_{i=0}^{k-1} C(n-c_\theta, i) \cdot C(c_\theta, k-i)}{C(n, k)}$$ where $c_\theta$ counts candidates satisfying criterion $\theta$ (Func, Sec, or Func-Sec).

Static Analysis/Dataset Metrics (Shahid et al., 24 Nov 2025, Haddad et al., 2023, Aghaei et al., 2020)

• Vulnerability Density (VD):

$$\mathrm{VD} = \frac{\sum_{\mathrm{cwe}\in\mathrm{CWESet}} \mathrm{occurrences}(\mathrm{cwe})}{\mathrm{LoC}/1000}$$

• Model ranking, precision, recall, F1, MRR, MAP@k, NDCG@k: Used for CVE-to-CWE mapping evaluation.
• CASTLE Score (Dubniczky et al., 12 Mar 2025): Compound metric rewarding correct detection of Top-25 CWEs, penalizing false positives, and balancing coverage with specificity.

4. Empirical Findings and Comparative Results

Key results from recent studies:

• Secure code generation remains challenging: Pretrained LLMs exhibit much lower Func-Sec@1 rates (≤15%) compared to Func@1, indicating the difficulty of producing code that is both functionally correct and secure (Li et al., 8 Oct 2025).
• Instruction tuning and secure prompts help, but gaps remain: Secure-Instruct tuning elevates CodeLlama-7B’s Func-Sec@1 on CWEval from 8.9% to 22.9% (+14.0 points). For Mistral-7B, improvements are similarly substantial but do not reach levels suggesting robust security across the board (Li et al., 8 Oct 2025). Secure-assistant prompts (e.g., in (Shahid et al., 24 Nov 2025)) reduce vulnerability density by ~12% on average, but no setting eliminates critical flaws entirely.
• Prevalence of "functional but insecure" code: Across benchmarks, many LLM outputs pass all functional tests yet fail the corresponding security tests. CWEval’s metrics, particularly func-sec@k, expose this discrepancy in ways static analysis or separated benchmarks do not (Peng et al., 14 Jan 2025).
• CWE distribution: Memory-safety related CWEs account for over half of vulnerabilities found in LLM-generated code (notably CWE-120, CWE-122, CWE-252, CWE-253, CWE-787, and CWE-401) (Shahid et al., 24 Nov 2025).
• Empirical comparisons: In CASTLE, LLMs outperformed all traditional static analyzers and formal verification tools on microbenchmarks for line-level CWE detection; however, hallucinations and recall sharply decline with growing input size and complexity (Dubniczky et al., 12 Mar 2025).
• CVE mapping accuracy: Hierarchical neural network models (e.g., ThreatZoom) achieve fine-grained CVE→CWE accuracy up to 92% (NVD) and 75% (MITRE), highlighting the potential for robust automated mapping (Aghaei et al., 2020), with fine-tuned SBERT also reaching MRR ≈ 0.91 (Haddad et al., 2023).

5. Methodological Insights and Limitations

• Manual verification is essential: All major CWEval datasets employ hand-reviewed task and oracle design to guarantee both functional appropriateness and security coverage, addressing weaknesses in prior benchmarks (Peng et al., 14 Jan 2025, Li et al., 8 Oct 2025).
• Dynamic oracles outperform static analysis: By executing code, CWEval can evaluate semantic variants missed by rule-based analyzers and reduce false reporting on secure but atypical implementations (Peng et al., 14 Jan 2025).
• Coverage is bounded by manual effort: Current versions of CWEval-bench and similar datasets are limited in CWE/task coverage compared to the full space of security weaknesses (Peng et al., 14 Jan 2025). Extension to new vulnerability classes, languages, or concurrency/side-channel issues awaits automation or further community investment.
• Security training can cause alignment tax: Overly aggressive security alignment (e.g., SafeCoder-style fine-tuning) risks degrading LLM functionality by incentivizing task refusal, not just vulnerability reduction (Peng et al., 14 Jan 2025).
• Static/dynamic tool combinations suggested: No single method uncovers all classes of CWEs. Synergistic toolchains (e.g., static analyzer + LLM) are advised for comprehensive coverage, with CASTLE Score as a selection criterion (Dubniczky et al., 12 Mar 2025).

6. Broader Implications and Future Directions

• Advancing secure code generation: CWEval’s dynamic, outcome-oriented methodology informs both academic research and industry best practices, providing metrics that force attention on real-world exploitability, not just syntactic compliance (Peng et al., 14 Jan 2025).
• Automated vulnerability mapping and remediation: Machine learning techniques (hierarchical networks, semantic rankers) have begun to automate CVE→CWE conversion and can power tools that prioritize mitigation of high-impact weaknesses (Aghaei et al., 2020, Haddad et al., 2023).
• Integrating with developer workflows: Integration of outcome-driven CWEval evaluations, secure prompt engineering, static/dynamic analyses, and human-in-the-loop annotation form the basis of actionable CI/CD security review—and are recommended as a multilayered defense (Shahid et al., 24 Nov 2025).
• Future expansion: Growth directions include automated oracle/task generation, broader language/framework support, adversarial input testing, coverage-tied metric reporting, and deeper links with security ontologies (e.g., ATT&CK, CAPEC) (Peng et al., 14 Jan 2025, Aghaei et al., 2020).

---

CWEval’s rigorous, executable, and CWE-grounded evaluation paradigm is establishing itself as the reference framework for empirical security assessment of automatically generated code, automated vulnerability mapping, and benchmarking of security analysis tools. Its methodological emphasis on combined functional and security validation, dynamic test oracles, and transparent, reproducible metrics has set a new standard for empirical research in software security evaluation (Peng et al., 14 Jan 2025, Li et al., 8 Oct 2025, Shahid et al., 24 Nov 2025, Aghaei et al., 2020, Haddad et al., 2023, Dubniczky et al., 12 Mar 2025).