Randomized block Kaczmarz with volume sampling: Momentum acceleration and efficient implementation

Abstract: The randomized block Kaczmarz (RBK) method is a widely utilized iterative scheme for solving large-scale linear systems. However, the theoretical analysis and practical effectiveness of this method heavily rely on a good row paving of the coefficient matrix. This motivates us to introduce a novel block selection strategy to the RBK method, called volume sampling, in which the probability of selection is proportional to the volume spanned by the rows of the selected submatrix. To further enhance the practical performance, we develop and analyze a momentum variant of the method. Convergence results are established and demonstrate the notable improvements in convergence factor of the RBK method brought by the volume sampling and the momentum acceleration. Furthermore, to efficiently implement the RBK method with volume sampling, we propose an efficient algorithm that enables volume sampling from a sparse matrix with sampling complexity that is only logarithmic in dimension. Numerical experiments confirm our theoretical results.