A tutorial comparing different covariate balancing methods with an application evaluating the causal effects of substance use treatment programs for adolescents

Abstract: Randomized controlled trials are the gold standard for measuring causal effects. However, they are often not always feasible, and causal treatment effects must be estimated from observational data. Observational studies do not allow robust conclusions about causal relationships unless statistical techniques account for the imbalance of pretreatment confounders across groups while key assumptions hold. Propensity score and balance weighting (PSBW) are useful techniques that aim to reduce the imbalances between treatment groups by weighting the groups to look alike on the observed confounders. There are many methods available to estimate PSBW. However, it is unclear a priori which will achieve the best trade-off between covariate balance and effective sample size. Moreover, it is critical to assess the validity of key assumptions required for robust estimation of the needed treatment effects, including the overlap and no unmeasured confounding assumptions. We present a step-by-step guide to covariate balancing strategies, including how to evaluate overlap, obtain estimates of PSBW, check for covariate balance, and assess sensitivity to unobserved confounding. We compare the performance of several estimation methods using a case study examining the relative effectiveness of substance use treatment programs and provide a user-friendly web application that can implement the proposed steps.