A Randomized Block-Coordinate Primal-Dual Method for Large-scale Stochastic Saddle Point Problems

Abstract: We consider (stochastic) convex-concave saddle point (SP) problems with high-dimensional decision variables, arising in various machine learning problems. To contend with the challenges in computing full gradients, we employ a randomized block-coordinate primal-dual scheme in which randomly selected primal and dual blocks of variables are updated. We consider both deterministic and stochastic settings, where deterministic partial gradients and their randomly sampled estimates are used, respectively, at each iteration. We investigate the convergence of the proposed method under different blocking strategies and provide the corresponding complexity results. While the best-known complexity result for deterministic primal-dual methods using full gradients is $\mathcal O(\max{M,N}^2/\varepsilon)$ where $M$ and $N$ denote the number of primal and dual blocks, respectively, we show that our proposed randomized block-coordinate method can achieve an improved convergence rate of $\mathcal O(MN/\varepsilon)$. Moreover, for the stochastic setting where a mini-batch sample gradient is utilized, we show an optimal oracle complexity of $\tilde{\mathcal{O}}(M^2N^2/\varepsilon^2)$ through acceleration. Finally, almost sure convergence of the iterate sequence to a saddle point is established.