Confidence Sets for $Z$-estimation Problems using Self-normalization

Abstract: Many commonly used statistical estimators are derived from optimization problems. This includes maximum likelihood estimation, empirical risk minimization, and so on. In many cases, the resulting estimators can be written as solutions to estimating equations, sometimes referred to as $Z$-estimators. Asymptotic normality for $Z$-estimators is a well-known result albeit when the dimension of the parameter is asymptotically smaller than the square root of the sample size. This hinders statistical inference when the dimension is "large." In this paper, we propose a self-normalization-based confidence set bypassing the asymptotic normality results. The proposed method is valid in the full range of dimensions growing smaller than the sample size (ignoring logarithmic factors) and asymptotically matches the asymptotic normality based confidence sets when asymptotic normality holds. Our proposal represents the first such general construction of confidence sets in the full range of consistency of $Z$-estimators.