The Güler-type acceleration for proximal gradient, linearized augmented Lagrangian and linearized alternating direction method of multipliers

Abstract: In this paper, we introduce the Güler-type acceleration technique and utilize it to propose three acceleration algorithms: the Güler-type accelerated proximal gradient method (GPGM), the Güler-type accelerated linearized augmented Lagrangian method (GLALM) and the Güler-type accelerated linearized alternating direction method of multipliers (GLADMM). The key idea behind these algorithms is to fully leverage the information of negative term \bm{$-|x^k-\hat{x}^{k-1}|^2$} in order to design the extrapolation step. This concept of using negative terms to improve acceleration can be extended to other algorithms as well. Moreover, the proposed GLALM and GLADMM enable simultaneous acceleration of both primal and dual variables. Additionally, GPGM and GLALM achieve the same convergence rate of $O(\frac{1}{k^2})$ with some existing results. Although GLADMM achieves the same total convergence rate of $O(\frac{1}{N})$ as in existing results, the partial convergence rate is improved from $O(\frac{1}{N^{3/2}})$ to $O(\frac{1}{N^2})$. To validate the effectiveness of our algorithms, we conduct numerical experiments on various problem instances, including the $\ell_1$ regularized logistic regression, quadratic programming, and compressive sensing. The experimental results indicate that our algorithms outperform existing methods in terms of efficiency. This also demonstrates the potential of the stochastic algorithmic versions of these algorithms in application areas such as statistics, machine learning, and data mining. Finally, it is worth noting that this paper aims to introduce how Güler's acceleration technique can be applied to gradient-based algorithms and to provide a unified and concise framework for their construction.