Anti-Phishing Training Does Not Work: A Large-Scale Empirical Assessment of Multi-Modal Training Grounded in the NIST Phish Scale

Abstract: Social engineering attacks using email, commonly known as phishing, are a critical cybersecurity threat. Phishing attacks often lead to operational incidents and data breaches. As a result, many organizations allocate a substantial portion of their cybersecurity budgets to phishing awareness training, driven in part by compliance requirements. However, the effectiveness of this training remains in dispute. Empirical evidence of training (in)effectiveness is essential for evidence-based cybersecurity investment and policy development. Despite recent measurement studies, two critical gaps remain in the literature: (1) we lack a validated measure of phishing lure difficulty, and (2) there are few comparisons of different types of training in real-world business settings. To fill these gaps, we conducted a large-scale study ($N = 12{,}511$) of phishing effectiveness at a US-based financial technology (``fintech'') firm. Our two-factor design compared the effect of treatments (lecture-based, interactive, and control groups) on subjects' susceptibility to phishing lures of varying complexity (using the NIST Phish Scale). The NIST Phish Scale successfully predicted behavior (click rates: 7.0\% easy to 15.0\% hard emails, p $<$ 0.001), but training showed no significant main effects on clicks (p = 0.450) or reporting (p = 0.417). Effect sizes remained below 0.01, indicating little practical value in any of the phishing trainings we deployed. Our results add to the growing evidence that phishing training is ineffective, reinforcing the importance of phishing defense-in-depth and the merit of changes to processes and technology to reduce reliance on humans, as well as rebuking the training costs necessitated by regulatory requirements.