Double Machine Learning for Conditional Moment Restrictions: IV Regression, Proximal Causal Learning and Beyond

Abstract: Solving conditional moment restrictions (CMRs) is a key problem considered in statistics, causal inference, and econometrics, where the aim is to solve for a function of interest that satisfies some conditional moment equalities. Specifically, many techniques for causal inference, such as instrumental variable (IV) regression and proximal causal learning (PCL), are CMR problems. Most CMR estimators use a two-stage approach, where the first-stage estimation is directly plugged into the second stage to estimate the function of interest. However, naively plugging in the first-stage estimator can cause heavy bias in the second stage. This is particularly the case for recently proposed CMR estimators that use deep neural network (DNN) estimators for both stages, where regularisation and overfitting bias is present. We propose DML-CMR, a two-stage CMR estimator that provides an unbiased estimate with fast convergence rate guarantees. We derive a novel learning objective to reduce bias and develop the DML-CMR algorithm following the double/debiased machine learning (DML) framework. We show that our DML-CMR estimator can achieve the minimax optimal convergence rate of $O(N^{-1/2})$ under parameterisation and mild regularity conditions, where $N$ is the sample size. We apply DML-CMR to a range of problems using DNN estimators, including IV regression and proximal causal learning on real-world datasets, demonstrating state-of-the-art performance against existing CMR estimators and algorithms tailored to those problems.

• The paper proposes the DML-CMR estimator, using Neyman orthogonality to correct bias from two-stage DNN regressions.
• The estimator achieves the minimax optimal convergence rate of O(N^(-1/2)), outperforming leading methods in IV regression and causal tasks.
• The study extends double machine learning to proximal causal learning, offering a robust solution for high-dimensional econometric challenges.

Double Machine Learning for Conditional Moment Restrictions: IV Regression, Proximal Causal Learning, and Beyond

The paper "Double Machine Learning for Conditional Moment Restrictions: IV Regression, Proximal Causal Learning, and Beyond" presents a novel approach to solving conditional moment restrictions (CMRs) using a double machine learning (DML) framework. The study addresses a critical aspect in statistics, causal inference, and econometrics, focusing on estimating a function via conditional moment equalities. The DML framework is employed to mitigate biases often seen in two-stage regression problems that use deep neural networks (DNNs) for estimation.

Key Contributions

The authors propose the DML-CMR estimator, which is designed to solve CMRs using a two-stage strategy that incorporates the DML framework. This framework is extended to provide unbiased estimates even when the first-stage DNN estimators introduce regularization and overfitting biases. A Neyman orthogonal score is developed to correct for such biases, providing strong convergence rate guarantees for the estimator.

The paper shows that DML-CMR can achieve the minimax optimal convergence rate of $O(N^{-1/2})$, where $N$ is the sample size, under certain regularity conditions. By framing the CMR problem with Neyman orthogonality, the authors ensure that small perturbations in nuisance parameter estimation do not significantly impact the estimation of the primary function of interest.

Methodology and Results

The paper's methodology revolves around refining the estimation process in two-stage regressions. In the first stage, nuisance parameters are learned using DNNs. This includes estimating conditional expectations necessary for the subsequent stage. The second stage involves the optimization of a score function, which is carefully designed to achieve Neyman orthogonality. This setup provides robustness against estimation inaccuracies in the first stage, ensuring the estimator retains optimal properties despite potential biases.

The efficacy of DML-CMR is demonstrated through empirical evaluations on a range of problems, including instrumental variable (IV) regression and proximal causal learning (PCL). For IV regression, the algorithm outperforms state-of-the-art methods such as Deep IV, DeepGMM, KIV, and DFIV in terms of mean squared error (MSE) on both synthetic and semi-synthetic datasets, including high-dimensional contexts. In PCL tasks, DML-CMR demonstrates superior performance over competing methods like CEVAE, PMMR, and NMMR.

Implications and Future Directions

This research provides significant implications for both theoretical and applied aspects of causal inference and econometrics. The introduction of the Neyman orthogonal score in the context of neural networks opens pathways to more reliable causal estimation frameworks, particularly in settings complicated by high-dimensionality and nonlinearity.

In practical terms, harnessing the strengths of DML in conjunction with modern machine learning techniques offers a flexible solution to tackle various econometric and statistical challenges. The approach's robust theoretical foundation also suggests potential applications in other areas of machine learning where model bias is a primary concern.

Future research might explore extensions of this framework to more complex models and settings, such as those involving time series or panel data. Additionally, further exploration of nonparametric forms of the function of interest within this framework could provide even greater flexibility and utility.

In conclusion, the paper offers a comprehensive advancement in the methodology of solving conditional moment restrictions by effectively merging machine learning capabilities with econometric rigor, enhancing the reliability of causal effect estimation in the presence of latent confounders.