On the probability of invalidating a causal inference due to limited external validity

Abstract: External validity is often questionable in empirical research, especially in randomized experiments due to the trade-off between internal validity and external validity. To quantify the robustness of external validity, one must first conceptualize the gap between a sample that is fully representative of the target population (i.e., the ideal sample) and the observed sample. Drawing on Frank & Min (2007) and Frank et al. (2013), I define such gap as the unobserved sample and intend to quantify its relationship with the null hypothesis statistical testing (NHST) in this study. The probability of invalidating a causal inference due to limited external validity, i.e., the PEV, is the probability of failing to reject the null hypothesis based on the ideal sample provided the null hypothesis has been rejected based on the observed sample. This study illustrates the guideline and the procedure of evaluating external validity with the PEV through an empirical example (i.e., Borman et al. (2008)). Specifically, one would be able to locate the threshold of the unobserved sample statistic that would make the PEV higher than a desired value and use this information to characterize the unobserved sample that would render external validity of the research in question less robust. The PEV is shown to be linked to statistical power when the NHST is thought to be based on the ideal sample.