Chasing Shadows: Pitfalls in LLM Security Research

Abstract: LLMs are increasingly prevalent in security research. Their unique characteristics, however, introduce challenges that undermine established paradigms of reproducibility, rigor, and evaluation. Prior work has identified common pitfalls in traditional machine learning research, but these studies predate the advent of LLMs. In this paper, we identify nine common pitfalls that have become (more) relevant with the emergence of LLMs and that can compromise the validity of research involving them. These pitfalls span the entire computation process, from data collection, pre-training, and fine-tuning to prompting and evaluation. We assess the prevalence of these pitfalls across all 72 peer-reviewed papers published at leading Security and Software Engineering venues between 2023 and 2024. We find that every paper contains at least one pitfall, and each pitfall appears in multiple papers. Yet only 15.7% of the present pitfalls were explicitly discussed, suggesting that the majority remain unrecognized. To understand their practical impact, we conduct four empirical case studies showing how individual pitfalls can mislead evaluation, inflate performance, or impair reproducibility. Based on our findings, we offer actionable guidelines to support the community in future work.

• The paper's main contribution is the comprehensive identification of nine key pitfalls, including model ambiguity and evaluation inflation in LLM security research.
• The study employs empirical analysis of 72 peer-reviewed papers, revealing high rates of data leakage, context truncation, and reproducibility issues.
• Actionable recommendations are provided, urging transparent model specifications and benchmark redesign to enhance the reliability of LLM security experiments.

Pitfalls and Limitations in LLM Security Research

Introduction

The paper "Chasing Shadows: Pitfalls in LLM Security Research" (2512.09549) delivers a rigorous meta-analysis of published work on the use of LLMs in security and software engineering contexts. The authors systematically enumerate nine distinct pitfalls, evaluate their prevalence across 72 peer-reviewed papers published between 2023 and 2024, and provide in-depth empirical case studies on the real-world consequences of these methodological issues. Their results expose the ubiquity of fundamental limitations, including the systematic absence of reproducibility, misinterpretation of benchmarks, and evaluation inflation—often unrecognized by the original contributors. The study emphasizes community-level awareness and actionable recommendations for researchers and practitioners operating within the LLM security domain.

Taxonomy of Pitfalls in LLM Security Research

The authors introduce a comprehensive taxonomy encapsulating nine problematic patterns impacting LLM-based security work:

• Data Poisoning via Internet Scraping: Insecure corpus construction exposes models to subtle data poisoning, especially given the scale and anonymity of platforms such as GitHub and Reddit.
• LLM-Generated Label Inaccuracy: Automated annotation using LLMs risks propagation of unvalidated or flawed labels, undermining experimental reliability.
• Data Leakage: Overlap between training and benchmark data—frequently undetected in opaque or proprietary model pipelines—leads to severe contamination and overestimation of generalization capabilities.
• Model Collapse through Synthetic Training Data: Recursive fine-tuning on LLM-generated outputs degrades distributional diversity and accumulates error, risking catastrophic collapse.
• Spurious Correlations: The vast parameter space and training scale facilitate memorization of shortcuts and non-causal artifacts, ultimately distorting measured performance and transferability.
• Context Truncation: Practical context window limits force input truncation, which can mask critical code or information in vulnerability detection and secure code generation.
• Prompt Sensitivity: Minor variations or mismatch in prompt format strongly affect outcomes, producing volatility and diminishing the reliability of comparative analysis.
• Proxy/Surrogate Fallacies: Erroneous generalization of findings between model classes, quantization levels, or versions, without empirical validation.
• Model Information Ambiguity: Incomplete specification of model identifiers, versions, access methods, or quantization prevents reproducibility and hinders precise interpretation.

Prevalence Assessment

The prevalence survey covers all 72 relevant papers published in major security and software engineering venues. Every paper exhibits at least one pitfall, and several pathologies (notably model information ambiguity, data leakage, and context truncation) are present in over 20% of studies. Only 15.7% of documented pitfalls were explicitly discussed, indicating systemic under-recognition of technical and methodological threats. Notably, the most prevalent pitfall, insufficient model specification (ambiguity in identifier, version, or quantization), affected 73.6% of the corpus but was not discussed in any publication.

The comparison across research topics revealed highest pitfall densities in vulnerability detection and repair, whereas fuzzing and secure code generation exhibited relatively fewer systematic failures.

Empirical Case Studies

To move beyond qualitative prevalence, the paper presents four rigorous empirical analyses quantifying practical impacts:

Model Information Ambiguity and Proxy Fallacies

Experiments on hate speech detection using different OpenAI GPT-4 snapshots and multiple open-source quantization settings demonstrate substantial shifts in precision, recall, and attack success rates. For example, accuracy dropped from 88.7% to 77.2% and recall plummeted from 82% to 41% when switching model instances, even with identical prompts and datasets. Robustness against prompt-based attacks depended critically on snapshot and quantization parameters, with attack success rates fluctuating by factors of three or more.

Data Leakage

Controlled fine-tuning of vulnerability detection models revealed near-linear increases in F1-score corresponding to leakage ratios; even 20% leakage elevated scores by 0.08–0.11, confirming that benchmark contamination can severely inflate performance metrics. Yet, targeted memorization probes on proprietary models found no evidence of test data reproduction in real-world settings, suggesting that leakage risks are instance-dependent and demand active investigation.

Context Truncation

Analysis of context windows in common vulnerability datasets revealed that, with standard 512-token contexts, nearly 50% of vulnerable functions are truncated, with 29% exceeding 1024 tokens. In concrete scenarios, the actual vulnerable code was outside the model’s input window, indicating that reliability of detection is undermined, and reported performance may be fundamentally misrepresentative.

Model Collapse

Iterative retraining on self-generated synthetic code, using Qwen2.5-Coder-0.5B-Instruct, demonstrated a monotonic increase in mean and variance of perplexity across generations, serving as a quantitative signal for model collapse.

Figure 1: Distribution of perplexities for training samples generated across multiple model generations; both mean and variance increase, indicating stability degradation through recursive synthetic training.

Practical and Theoretical Implications

The findings necessitate substantial protocol changes for LLM-focused research. In absence of transparent model specification, reproducibility is unattainable, rendering empirical comparisons between studies technically invalid. The risks of data leakage and context truncation undermine both experimental integrity and real-world deployment credibility, especially in security-critical applications such as vulnerability detection, penetration testing, and automated code repair. Model collapse and spurious correlation exploitation further threaten the longevity of generative pipelines, as iterative model retraining or automated annotation becomes increasingly prevalent in both research and industry.

The authors’ recommendations—detailed specification of model family, version, quantization, prompt structure, and explicit evaluation for data leakage—represent minimum requirements for technical rigor. They argue for treating pitfall avoidance as fundamental design constraints, with the community urged to adopt transparent, testable methodologies aligned with regulatory movements (e.g., EU AI Act).

From a theoretical perspective, the universality of these pitfalls challenges the reliability of both "state-of-the-art" performance claims and security claims regarding attack/defense efficacy. Future directions should include benchmark redesign with explicit leakage and contamination probes, model versioning standards, robust prompt and context ablation testing, and longitudinal studies on generative drift and collapse dynamics under operational retraining regimes.

Conclusion

This paper establishes that methodologically significant pitfalls are pervasive in LLM security research, typically underappreciated and often unaddressed. Strong empirical evidence is provided for the distortion of results via model ambiguity, data contamination, context truncation, and iterative synthetic data training. The practical upshot is that reported results in high-impact venues may substantially misrepresent security, robustness, and generalization. An effective path forward requires formalization, transparency, and constant audit of these pitfalls in both academic and production-level LLM pipelines.