Estimating and evaluating counterfactual prediction models

Abstract: Counterfactual prediction methods are required when a model will be deployed in a setting where treatment policies differ from the setting where the model was developed, or when a model provides predictions under hypothetical interventions to support decision-making. However, estimating and evaluating counterfactual prediction models is challenging because, unlike traditional (factual) prediction, one does not observe the full set of potential outcomes for all individuals. Here, we discuss how to fit or tailor a model to target a counterfactual estimand, how to assess the model's performance, and how to perform model and tuning parameter selection. We provide identifiability and estimation results for building a counterfactual prediction model and for multiple measures of counterfactual model performance including loss-based measures, the area under the receiver operating characteristics curve, and calibration. Importantly, our results allow valid estimates of model performance under counterfactual intervention even if the candidate model is misspecified, permitting a wider array of use cases. We illustrate these methods using simulation and apply them to the task of developing a statin-naive risk prediction model for cardiovascular disease.