RGDBEK: Randomized Greedy Double Block Extended Kaczmarz Algorithm with Hybrid Parallel Implementation and Applications

Abstract: Kaczmarz is one of the most prominent iterative solvers for linear systems of equations. Despite substantial research progress in recent years, the state-of-the-art Kaczmarz algorithms have not fully resolved the seesaw effect, a major impediment to convergence stability. Furthermore, while there have been advances in parallelizing the inherently sequential Kaczmarz method, no existing architecture effectively supports initialization-independent parallelism that fully leverages both CPU and GPU resources. This paper proposes the Randomized Greedy Double Block Extended Kaczmarz (RGDBEK) algorithm, a novel Kaczmarz approach designed for efficient large-scale linear system solutions. RGDBEK employs a randomized selection strategy for column and row blocks based on residual-derived probability distributions, thereby mitigating the traditional seesaw effect and enhancing convergence robustness. Theoretical analysis establishes linear convergence of the method under standard assumptions. Extensive numerical experiments on synthetic random matrices and real-world sparse matrices from the SuiteSparse collection demonstrate that RGDBEK outperforms existing Kaczmarz variants, including GRK, FDBK, FGBK, and GDBEK, in both iteration counts and computational time. In addition, a hybrid parallel CPU-GPU implementation utilizing optimized sparse matrix-vector multiplications via the state-of-the-art storage format improves scalability and performance on large sparse problems. Applications in finite element discretizations, image deblurring, and noisy population modeling demonstrate the algorithm's versatility and effectiveness. Future work will explore extending RGDBEK to tensor systems, optimizing parallel parameter selection, and reducing communication overhead to further enhance efficiency and applicability.