Transfer Learning with General Estimating Equations

Abstract: We consider statistical inference for parameters defined by general estimating equations under the covariate shift transfer learning. Different from the commonly used density ratio weighting approach, we undertake a set of formulations to make the statistical inference semiparametric efficient with simple inference. It starts with re-constructing the estimation equations to make them Neyman orthogonal, which facilitates more robustness against errors in the estimation of two key nuisance functions, the density ratio and the conditional mean of the moment function. We present a divergence-based method to estimate the density ratio function, which is amenable to machine learning algorithms including the deep learning. To address the challenge that the conditional mean is parametric-dependent, we adopt a nonparametric multiple-imputation strategy that avoids regression at all possible parameter values. With the estimated nuisance functions and the orthogonal estimation equation, the inference for the target parameter is formulated via the empirical likelihood without sample splittings. We show that the proposed estimator attains the semiparametric efficiency bound, and the inference can be conducted with the Wilks' theorem. The proposed method is further evaluated by simulations and an empirical study on a transfer learning inference for ground-level ozone pollution