Accelerating Proximal Gradient Descent via Silver Stepsizes

Abstract: Surprisingly, recent work has shown that gradient descent can be accelerated without using momentum -- just by judiciously choosing stepsizes. An open question raised by several papers is whether this phenomenon of stepsize-based acceleration holds more generally for constrained and/or composite convex optimization via projected and/or proximal versions of gradient descent. We answer this in the affirmative by proving that the silver stepsize schedule yields analogously accelerated rates in these settings. These rates are conjectured to be asymptotically optimal among all stepsize schedules, and match the silver convergence rate of vanilla gradient descent (Altschuler and Parrilo, 2024, 2025), namely $O(\varepsilon^{- \log_{\rho} 2})$ for smooth convex optimization and $O(\kappa^{\log_\rho 2} \log \frac{1}{\varepsilon})$ under strong convexity, where $\varepsilon$ is the precision, $\kappa$ is the condition number, and $\rho = 1 + \sqrt{2}$ is the silver ratio. The key technical insight is the combination of recursive gluing -- the technique underlying all analyses of gradient descent accelerated with time-varying stepsizes -- with a certain Laplacian-structured sum-of-squares certificate for the analysis of proximal point updates.

• The paper demonstrates that nonmonotonic silver stepsizes can accelerate proximal gradient descent for constrained and composite convex optimization.
• This approach achieves improved convergence rates, potentially reaching the optimal conjectured rate of O("") in certain smooth convex cases.
• The findings suggest that smart stepsize scheduling could offer an alternative to momentum-based methods for enhancing efficiency in complex optimization problems.

Accelerating Proximal Gradient Descent via Silver Stepsizes

The paper "Accelerating Proximal Gradient Descent via Silver Stepsizes" by Jinho Bok and Jason M. Altschuler presents an innovative approach to augment the performance of gradient descent methods through the strategic selection of stepsizes, specifically the "silver stepsize" schedule. The research broadens the scope of stepsize-based acceleration, previously understood mainly within the confines of unconstrained, smooth convex optimization, to more complex optimization tasks, such as constrained and composite convex optimization.

Overview of Achievements

The authors focus on projected gradient descent (GD) for constrained optimization and proximal gradient descent for composite convex optimization, both relevant in handling more intricate problems where constraints or nonsmoothness are present. They address the previously unresolved question of whether the acceleration benefits of stepsize scheduling, noted in basic GD, can extend to these broader contexts without introducing momentum or other enhancements beyond smart stepsize management.

The core achievement is the demonstration that silver stepsizes, a specific, nonmonotonic, time-varying stepsize sequence, can yield accelerated convergence rates. For instance, in unconstrained, smooth convex scenarios, silver stepsizes achieve the conjectured optimal convergence rate of $O(\varepsilon^{-\log_{\rho} 2})$, where $\rho$ is the silver ratio. For composite optimization, the authors affirm that proximal GD, when paired with the silver stepsizes, also achieves these enhanced rates, improving upon the traditional $O(\varepsilon^{-1})$ rate found with constant stepsizes. Notably, these findings extend the theoretical guarantees of stepsize-based acceleration to include scenarios with constraints or composite objectives.

Technical Contributions

Bok and Altschuler harness a meticulous analytical approach combining recursive gluing and sum-of-squares (SOS) polynomials to establish this acceleration. Recursive gluing leverages the recursive nature of stepsizes to systematically verify multi-step descent processes, vital for handling the time-varying attributes of schedule-based accelerations. The SOS framework offers certificates for these accelerated moves, ensuring stability and convergence in such modified algorithmic paths.

The researchers also address challenges in stepsize scheduling where interleaving projection or proximal operations traditionally impacts algorithmic behavior. They overcome issues of fundamental divergence observed in these contexts by incorporating Laplacian-structured arguments, ensuring that multi-step dependencies render the updated algorithms robust and virtually optimal within their scheduling characterization.

Implications and Forward-Looking Remarks

The implications of this work are extensive in the domain of large-scale optimization where the constraints and nonsmooth features are prevalent. Silver stepsizes' ability to accelerate variants of GD in such settings implies that practitioners may need to revisit their optimization protocols, replacing traditional momentum strategies with smarter stepsize schedules for enhanced efficiency.

The paper sets the stage for future exploration into blending advanced stepsize schedules with adaptive and nonadaptive procedures within other optimization schemas. An exploration into further empirical validation, especially within stochastic settings, could provide additional insights into how these schedules behave under uncertainty and data variance—a common theme in machine-learning applications.

As proximal methods extend beyond convex problems, addressing their adaptation to other structured problem subclasses and leveraging automated processes for optimal scheduling discovery using machine learning could propel advancements towards universally adaptive optimization techniques, underscoring a robust theoretical foundation as established in this paper.