Papers
Topics
Authors
Recent
Search
2000 character limit reached

Near-optimal Closed-loop Method via Lyapunov Damping for Convex Optimization

Published 16 Nov 2023 in math.OC, cs.LG, and math.DS | (2311.10053v2)

Abstract: We introduce an autonomous system with closed-loop damping for first-order convex optimization. While, to this day, optimal rates of convergence are almost exclusively achieved by non-autonomous methods via open-loop damping (e.g., Nesterov's algorithm), we show that our system, featuring a closed-loop damping, exhibits a rate arbitrarily close to the optimal one. We do so by coupling the damping and the speed of convergence of the system via a well-chosen Lyapunov function. By discretizing our system we then derive an algorithm and present numerical experiments supporting our theoretical findings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (35)
  1. Finite time stabilization of nonlinear oscillators subject to dry friction. In Nonsmooth Mechanics and Analysis, pages 289–304, 2006.
  2. An extension of the second order dynamical system that models Nesterov’s convex gradient method. Applied Mathematics & Optimization, 84:1687–1716, 2021.
  3. A second-order gradient-like dissipative dynamical system with Hessian-driven damping: Application to optimization and mechanics. Journal de mathématiques pures et appliquées, 81(8):747–779, 2002.
  4. Asymptotic stabilization of inertial gradient dynamics with time-dependent viscosity. Journal of Differential Equations, 263(9):5412–5458, 2017.
  5. From the Ravine method to the Nesterov method and vice versa: a dynamical system perspective. SIAM Journal on Optimization, 32(3):2074–2101, 2022.
  6. Fast convergence of inertial dynamics and algorithms with asymptotic vanishing damping. Mathematical Programming, 168:123–175, 2015.
  7. A dynamic approach to a proximal-Newton method for monotone inclusions in Hilbert spaces, with complexity O⁢(1/n2)𝑂1superscript𝑛2{O}(1/n^{2})italic_O ( 1 / italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ). Journal of Convex Analysis, 23:139–180, 2016a.
  8. Fast convex optimization via inertial dynamics with Hessian driven damping. Journal of Differential Equations, 261(10):5734–5783, 2016b.
  9. Rate of convergence of the Nesterov accelerated gradient method in the subcritical case α≤𝛼absent\alpha\leqitalic_α ≤ 3. ESAIM: Control, Optimisation and Calculus of Variations, 25:2, 2019.
  10. Fast optimization via inertial dynamics with closed-loop damping. Journal of the European Mathematical Society, 25(5):1985–2056, 2022.
  11. Fast convex optimization via closed-loop time scaling of gradient dynamics. arXiv preprint arXiv:2301.00701, 2023.
  12. Optimal rate of convergence of an ODE associated to the fast gradient descent schemes for b>0𝑏0b>0italic_b > 0. HAL Preprint hal-01547251, 2017.
  13. Optimal convergence rates for Nesterov acceleration. SIAM Journal on Optimization, 29(4):3131–3153, 2019.
  14. Stochastic approximations and differential inclusions. SIAM Journal on Control and Optimization, 44(1):328–348, 2005.
  15. Subgradient methods. Lecture notes of EE392o, Stanford University, 2003.
  16. Asymptotics for some semilinear hyperbolic equations with non-autonomous damping. Journal of Differential Equations, 252(1):294–322, 2012.
  17. On the long time behavior of second order differential equations with asymptotically small dissipation. Transactions of the American Mathematical Society, 361(11):5983–6017, 2009.
  18. On the convergence of the iterates of the “fast iterative shrinkage/thresholding algorithm”. Journal of Optimization theory and Applications, 166:968–982, 2015.
  19. George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303–314, 1989.
  20. David M Grobman. Homeomorphism of systems of differential equations. Doklady Akademii Nauk SSSR, 128(5):880–881, 1959.
  21. Philip Hartman. A lemma in the theory of structural stability of differential equations. Proceedings of the American Mathematical Society, 11(4):610–620, 1960.
  22. Asymptotics for a second-order differential equation with nonautonomous damping and an integrable source term. Applicable Analysis, 94(2):435–443, 2015.
  23. A control-theoretic perspective on optimal high-order optimization. Mathematical Programming, 195(1):929–975, 2022.
  24. Lennart Ljung. Analysis of recursive stochastic algorithms. IEEE transactions on automatic control, 22(4):551–575, 1977.
  25. Ramzi May. Long time behavior for a semilinear hyperbolic equation with asymptotically vanishing damping term and convex potential. Journal of Mathematical Analysis and Applications, 430(1):410–416, 2015.
  26. Ramzi May. Asymptotic for a second-order evolution equation with convex potential and vanishing damping term. Turkish Journal of Mathematics, 41(3):681–685, 2017.
  27. Yurii Nesterov. A method of solving a convex programming problem with convergence rate O(1k2)1superscript𝑘2\left(\frac{1}{k^{2}}\right)( divide start_ARG 1 end_ARG start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ). In Doklady Akademii Nauk, volume 269(3), pages 543–547, 1983.
  28. Peter Philip. Ordinary differential equations, lecture notes, 2023. URL https://www.math.lmu.de/~philip/publications/lectureNotes/philipPeter_ODE.pdf. [Online; accessed 17.07.2023].
  29. Boris T. Polyak. Minimization of unsmooth functionals. USSR Computational Mathematics and Mathematical Physics, 9(3):14–29, 1969.
  30. B.T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1–17, 1964.
  31. Evolutionary Equations: Picard’s Theorem for Partial Differential Equations, and Applications. Springer Nature, 2022.
  32. Understanding the acceleration phenomenon via high-resolution differential equations. Mathematical Programming, 195(1):79–148, 2021.
  33. A differential equation for modeling Nesterov’s accelerated gradient method: Theory and insights. In Advances in Neural Information Processing Systems (NeurIPS), volume 27, pages 1–9. Curran Associates, Inc., 2014.
  34. A variational perspective on accelerated methods in optimization. Proceedings of the National Academy of Sciences, 113(47):E7351–E7358, 2016.
  35. A Lyapunov analysis of accelerated methods in optimization. Journal of Machine Learning Research, 22(1):5040–5073, 2021.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up