Robust learning and complexity dependent bounds for regularized problems

Abstract: We study Regularized Empirical Risk Minimizers (RERM) and minmax Median-Of-Means (MOM) estimators where the regularization function $\phi(\cdot)$ is an even convex function. We obtain bounds on the $L_2$-estimation error and the excess risk that depend on $\phi(f^*)$, where $f^*$ is the minimizer of the risk over a class $F$. The estimators are based on loss functions that are both Lipschitz and convex. Results for the RERM are derived under weak assumptions on the outputs and a sub-Gaussian assumption on the class ${ (f-f^*)(X), f \in F }$. Similar results are shown for minmax MOM estimators in a close setting where outliers may corrupt the dataset and where the class ${ (f-f^*)(X), f \in F }$ is only supposed to satisfy weak moment assumptions, relaxing the sub-Gaussian and the i.i.d hypothesis necessary for RERM. The analysis of RERM and minmax MOM estimators with Lipschitz and convex loss funtions is based on a weak local Bernstein Assumption. We obtain two "meta theorems" that we use to study linear estimators regularized by the Elastic Net. We also examine Support Vector Machines (SVM), where no sub-Gaussian assumption is required and when the target $Y$ can be heavy-tailed, improving the existing literature.