Newton method for $\ell_0$-regularized optimization

Abstract: As a tractable approach, regularization is frequently adopted in sparse optimization. This gives rise to the regularized optimization, aiming at minimizing the $\ell_0$ norm or its continuous surrogates that characterize the sparsity. From the continuity of surrogates to the discreteness of $\ell_0$ norm, the most challenging model is the $\ell_0$-regularized optimization. To conquer this hardness, there is a vast body of work on developing numerically effective methods. However, most of them only enjoy that either the (sub)sequence converges to a stationary point from the deterministic optimization perspective or the distance between each iterate and any given sparse reference point is bounded by an error bound in the sense of probability. In this paper, we develop a Newton-type method for the $\ell_0$-regularized optimization and prove that the generated sequence converges to a stationary point globally and quadratically under the standard assumptions, theoretically explaining that our method is able to perform surprisingly well.