Redefining Populations of Inference for Generalizations from Small Studies

Abstract: With the growth in experimental studies in education, policymakers and practitioners are interested in understanding not only what works, but for whom an intervention works. This interest in the generalizability of a study's findings has benefited from advances in statistical methods that aim to improve generalizations, particularly when the original study sample is not randomly selected. A challenge, however, is that generalizations are frequently based on small study samples. Limited data affects both the precision and bias of treatment impact estimates, calling into question the validity of generalizations. This study explores the extent to which redefining the inference population is a useful tool to improve generalizations from small studies. We discuss two main frameworks for redefining populations and apply the methods to an empirical example based on a completed cluster randomized trial in education. We discuss the implications of various methods to redefine the population and conclude with guidance and some recommendations for practitioners interested in using redefinition.