Generalizability with ignorance in mind: learning what we do (not) know for archetypes discovery

Abstract: When studying policy interventions, researchers are often interested in two related goals: i) learning for which types of individuals the program has the largest effects (heterogeneity) and ii) understanding whether those patterns of treatment effects have predictive power across environments (generalizability). To that end, we develop a framework to learn from the data how to partition observations into groups of individual and environmental characteristics whose effects are generalizable for others - a set of generalizable archetypes. Our view is that implicit in the task of archetypal discovery is detecting those contexts where effects do not generalize and where researchers should collect more evidence before drawing inference on treatment effects. We introduce a method that jointly estimates when and how a prediction can be formed and when, instead, researchers should admit ignorance and elicit further evidence before making predictions. We provide both a decision-theoretic and Bayesian foundation of our procedure. We derive finite-sample (frequentist) regret guarantees, asymptotic theory for inference, and discuss computational properties. We illustrate the benefits of our procedure over existing alternatives that would fail to admit ignorance and force pooling across all units by re-analyzing a multifaceted program targeted towards the poor across six different countries.