Untangling Sample and Population Level Estimands in Bayesian Causal Inference

Abstract: Bayesian inference for causal estimands has been growing in popularity, however many misconceptions and implementation errors arise from conflating sample and population-level estimands. We have anecdotally witnessed these at conference talks, in the course of peer review service, and even in published and arXiv-ed papers. Our goal here is to elucidate the crucial differences between sample and population-level quantities when it comes to identification, modeling, Bayesian computation, and interpretation. For example, common sample-level estimands require cross-world assumptions for identification, whereas common population-level estimands do not. Similarly, the former requires explicit imputation of counterfactuals from their joint posterior, whereas the latter typically only requires a posterior distribution over parameters. We start by defining some examples of both types of estimands, then discuss the full joint posterior over all unknowns (both missing counterfactuals and population distribution parameters). We continue to outline how inference for different estimands are derived from different marginals of this joint posterior. Because the differences are conceptually subtle but can be practically substantial, we provide an illustration of using synthetic data in Stan. We also provide a detailed appendix with derivations and computational tips along with a discussion of common implementation errors. The overarching message here is to always engage in first-principles thinking about which marginal of the joint posterior is of interest in a particular application, then follow the strict logic of Bayes' theorem and probability to avoid common implementation errors.