Block Coordinate Descent Methods for Structured Nonconvex Optimization with Nonseparable Constraints: Optimality Conditions and Global Convergence

Abstract: Coordinate descent algorithms are widely used in machine learning and large-scale data analysis due to their strong optimality guarantees and impressive empirical performance in solving non-convex problems. In this work, we introduce Block Coordinate Descent (BCD) method for structured nonconvex optimization with nonseparable constraints. Unlike traditional large-scale Coordinate Descent (CD) approaches, we do not assume the constraints are separable. Instead, we account for the possibility of nonlinear coupling among them. By leveraging the inherent problem structure, we propose new CD methods to tackle this specific challenge. Under the relatively mild condition of locally bounded non-convexity, we demonstrate that achieving coordinate-wise stationary points offer a stronger optimality criterion compared to standard critical points. Furthermore, under the Luo-Tseng error bound conditions, our BCD methods exhibit Q-linear convergence to coordinate-wise stationary points or critical points. To demonstrate the practical utility of our methods, we apply them to various machine learning and signal processing models. We also provide the geometry analysis for the models. Experiments on real-world data consistently demonstrate the superior objective values of our approaches compared to existing methods.