An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls

Abstract: We introduce new inference procedures for counterfactual and synthetic control methods for policy evaluation. We recast the causal inference problem as a counterfactual prediction and a structural breaks testing problem. This allows us to exploit insights from conformal prediction and structural breaks testing to develop permutation inference procedures that accommodate modern high-dimensional estimators, are valid under weak and easy-to-verify conditions, and are provably robust against misspecification. Our methods work in conjunction with many different approaches for predicting counterfactual mean outcomes in the absence of the policy intervention. Examples include synthetic controls, difference-in-differences, factor and matrix completion models, and (fused) time series panel data models. Our approach demonstrates an excellent small-sample performance in simulations and is taken to a data application where we re-evaluate the consequences of decriminalizing indoor prostitution. Open-source software for implementing our conformal inference methods is available.

An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls

The paper by Chernozhukov et al. presents a new approach to inferential methods in policy evaluation using counterfactual and synthetic control methods. The authors propose a novel method that systematically recasts causal inference problems into counterfactual prediction and structural break testing problems, adopting insights from conformal prediction and permutation-based inference techniques. This methodology is designed to be robust and discernible, offering validity even under weak conditions and is flexible to accommodate a variety of high-dimensional estimators.

Central to this proposed approach is the way the authors employ permutation inference to test hypotheses about the trajectory of policy effects. The innovative aspect lies in estimating counterfactual mean outcomes without requiring exhaustive data or stringent assumptions, thus maintaining robustness in analysis. This flexibility is crucial, as it permits the application of their method across different previously established techniques in policy analysis like synthetic controls, difference-in-differences, factor models, and time-series models.

The paper delineates how the proposed method remains valid under two settings:

1. Estimator Consistency and Stationary Errors: When estimators are consistent and the errors follow a stationary, weakly dependent process, the method holds its validity. This condition allows their method to be applied to popular existing methods by verifying minimal consistency.
2. Estimator Stability in Stationary Data: When estimator consistency cannot be assured due to model misspecification, the authors introduce the concept of estimator stability— a property indicating that estimators are not overly sensitive to small perturbations in data. This stability allows for model robustness even under the possibility of inconsistency, as long as the data are stationary.

The authors provide finite sample bounds for the performance of their method, ensuring asymptotic validity as sample sizes increase. This non-asymptotic nature of theoretical underpinning is particularly beneficial in practical applications where sample sizes might be limited, common in synthetic control scenarios.

Additional noteworthy contributions include the development of constrained Lasso as a flexible penalized regression model without the need for complex tuning parameters, offering a unifying framework that overlaps both synthetic control and difference-in-differences methods. Furthermore, they achieve consistency results for synthetic control estimators within the paradigm of many control units, an area that encountered difficulties in earlier research endeavors.

Practical implications of this work are vast, particularly in fields such as economics, public policy, and social sciences, where robust inference in small sample settings is frequently required. Theoretically, the paper challenges existing paradigms on inference assumptions, advocating for methods that minimize reliance on strict assumptions about data-generating processes. This pursuit can pave the way for future developments in statistical methodologies, especially under high-dimensional settings in counterfactual analysis.

Given the extensive theoretical and methodological advancements proposed, this work lays a solid foundation for future research on inference in high-dimensional and synthetic control settings. It prompts the exploration of similar robust algorithms that further widen the applicability and ease of causal analysis in complex and varied practical settings.