Randomized double and triple Kaczmarz for solving extended normal equations

Abstract: The randomized Kaczmarz algorithm has received considerable attention recently because of its simplicity, speed, and the ability to approximately solve large-scale linear systems of equations. In this paper we propose randomized double and triple Kaczmarz algorithms to solve extended normal equations of the form $\bf A^\top Ax=A^\top b-c$. The proposed algorithms avoid forming $\bf A^\top A$ explicitly and work for {\it arbitrary} $\mbf A\in\mbbr^{m\times n}$ (full rank or rank deficient, $m\geq n$ or $m<n$). {\it Tight} upper bounds showing exponential convergence in the mean square sense of the proposed algorithms are presented and numerical experiments are given to illustrate the theoretical results.