How Balanced Should Causal Covariates Be?

Abstract: Covariate balancing is a popular technique for controlling confounding in observational studies. It finds weights for the treatment group which are close to uniform, but make the group's covariate means (approximately) equal to those of the entire sample. A crucial question is: how approximate should the balancing be, in order to minimize the error of the final estimate? Current guidance is derived from heuristic or asymptotic analyses, which are uninformative when the size of the sample is small compared to the number of covariates. This paper presents the first rigorous, nonasymptotic analysis of covariate balancing; specifically, we use PAC-Bayesian techniques to derive valid, finite-sample confidence intervals for the treatment effect. More generally, we prove these guarantees for a flexible form of covariate balancing where the regularization parameters weighting the tradeoff between bias (imbalance) and variance (divergence from uniform) are optimized, not fixed. This gives rise to a new balancing algorithm which empirically delivers superior adaptivity. Our overall contribution is to make covariate balancing a more reliable method for causal inference.