Lipschitz minimization and the Goldstein modulus
Abstract: Goldstein's 1977 idealized iteration for minimizing a Lipschitz objective fixes a distance - the step size - and relies on a certain approximate subgradient. That "Goldstein subgradient" is the shortest convex combination of objective gradients at points within that distance of the current iterate. A recent implementable Goldstein-style algorithm allows a remarkable complexity analysis (Zhang et al. 2020), and a more sophisticated variant (Davis and Jiang, 2022) leverages typical objective geometry to force near-linear convergence. To explore such methods, we introduce a new modulus, based on Goldstein subgradients, that robustly measures the slope of a Lipschitz function. We relate near-linear convergence of Goldstein-style methods to linear growth of this modulus at minimizers. We illustrate the idea computationally with a simple heuristic for Lipschitz minimization.
- A unified analysis of descent sequences in weakly convex optimization, including convergence rates for bundle methods. SIAM Journal on Optimization, 33:89–115, 2023.
- N. Boumal. An Introduction to Optimization on Smooth Manifolds. Cambridge University Press, Cambridge, 2023.
- F.H. Clarke. Optimization and Nonsmooth Analysis. Wiley Interscience, New York, 1983.
- A gradient sampling method with complexity guarantees for lipschitz functions in high and low dimensions. In NeurIPS Proceedings, 2022.
- D. Davis and Liwei Jiang. A nearly linearly convergent first-order method for nonsmooth functions with quadratic growth. Found. Comput. Math., to appear, 2024. arXiv:2205.00064v3.
- D. Drusvyatskiy and A.S. Lewis. Optimality, identifiability, and sensitivity. Math. Program., 147:467–498, 2014.
- D. Drusvyatskiy and A.S. Lewis. Error bounds, quadratic growth, and linear convergence of proximal methods. Preprint arXiv:1602.06661, 2016.
- A.A. Goldstein. Optimization of Lipschitz continuous functions. Math. Programming, 13:14–22, 1977.
- E. Hazan and S. Kale. Beyond the regret minimization barrier: optimal algorithms for stochastic strongly-convex optimization. J. Mach. Learn. Res., 15:2489–2512, 2014.
- A.D Ioffe. Variational Analysis of Regular Mappings. Springer US, 2017.
- Deterministic nonsmooth nonconvex optimization. In Proceedings of Machine Learning Research, volume 195, pages 1–28, 2023.
- On the complexity of deterministic nonsmooth and nonconvex optimization. arXiv:2209.12463, 2022.
- The cost of nonconvexity in deterministic nonsmooth optimization. Mathematics of Operations Research, doi.org/10.1287/moor.2022.0289, 2023.
- G. Kornowski and O. Shamir. On the complexity of finding small subgradients in nonsmooth optimization. arXiv:2209.10346, 2022.
- Identifiability, the KL property in metric spaces, and subgradient curves. Fourndations of Computational Mathematics, 2024. To appear.
- A.S. Lewis and S. Zhang. Partial smoothness, tilt stability, and generalized Hessians. SIAM J. Optim., 23(1):74–94, 2013.
- J. Nocedal and S.J. Wright. Numerical Optimization. Springer Series in Operations Research and Financial Engineering. Springer, New York, second edition, 2006.
- S.M. Robinson. Linear convergence of epsilon-subgradient descent methods for a class of convex functions. Math. Program., 86:41–50, 1999.
- Lai Tian and Anthony Man-Cho So. Computing Goldstein (ϵ,δ)italic-ϵ𝛿(\epsilon,\delta)( italic_ϵ , italic_δ )-stationary points of Lipschitz functions in O~(ϵ−3δ−1)~𝑂superscriptitalic-ϵ3superscript𝛿1\widetilde{O}(\epsilon^{-3}\delta^{-1})over~ start_ARG italic_O end_ARG ( italic_ϵ start_POSTSUPERSCRIPT - 3 end_POSTSUPERSCRIPT italic_δ start_POSTSUPERSCRIPT - 1 end_POSTSUPERSCRIPT ) iterations via random conic perturbation. arxiv.org/abs/2112.09002, 2021.
- Complexity of finding stationary points of nonconvex nonsmooth functions. In ICML Proceedings, 2020.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
