Asymptotic behavior of unregularized and ridge-regularized high-dimensional robust regression estimators : rigorous results

Abstract: We study the behavior of high-dimensional robust regression estimators in the asymptotic regime where $p/n$ tends to a finite non-zero limit. More specifically, we study ridge-regularized estimators, i.e $\widehat{\beta}=\text{argmin}{\beta \in \mathbb{R}^p} \frac{1}{n}\sum{i=1}^n \rho(\varepsilon_i-X_i' \beta)+\frac{\tau}{2}\lVert\beta\rVert^2$. In a recently published paper, we had developed with collaborators probabilistic heuristics to understand the asymptotic behavior of $\widehat{\beta}$. We give here a rigorous proof, properly justifying all the arguments we had given in that paper. Our proof is based on the probabilistic heuristics we had developed, and hence ideas from random matrix theory, measure concentration and convex analysis. While most the work is done for $\tau>0$, we show that under some extra assumptions on $\rho$, it is possible to recover the case $\tau=0$ as a limiting case. We require that the $X_i$'s be i.i.d with independent entries, but our proof handles the case where these entries are not Gaussian. A 2-week old paper of Donoho and Montanari [arXiv:1310.7320] studied a similar problem by a different method and with a different point of view. At this point, their interesting approach requires Gaussianity of the design matrix.