Stochastic Gradient Descent Revisited

Abstract: Stochastic gradient descent (SGD) has been a go-to algorithm for nonconvex stochastic optimization problems arising in machine learning. Its theory however often requires a strong framework to guarantee convergence properties. We hereby present a full scope convergence study of biased nonconvex SGD, including weak convergence, function-value convergence and global convergence, and also provide subsequent convergence rates and complexities, all under relatively mild conditions in comparison with literature.

• The paper presents a novel framework for analyzing biased SGD in nonconvex settings without requiring bounded iterates.
• The paper derives specific convergence rates under local and global Łojasiewicz conditions for both gradient and function-value convergence.
• The paper quantifies computational complexity and guarantees global convergence to stationary points, informing practical deep learning optimization.

A Comprehensive Study of Convergence in Biased Stochastic Gradient Descent

The paper "Stochastic Gradient Descent Revisited" offers a meticulous examination of convergence properties for biased variants of Stochastic Gradient Descent (SGD) in nonconvex settings, expanding beyond traditional frameworks that predominately address convex cases. It establishes convergence rates, conditions for global convergence, and complexities using mild conditions relative to existing literature.

Convergence Modes and Conditions

Three main modes of convergence are explored: weak convergence, function-value convergence, and global convergence. Weak convergence involves the vanishing of gradients at iterated points almost surely. Function-value convergence pertains to the almost sure convergence of objective function values along the trajectory. Global convergence indicates convergence of iterates to a stationary point.

The paper introduces an analysis framework using the Łojasiewicz condition, a popular tool to study convergence in nonconvex optimization. Under this condition, the paper derives specific rates of convergence for biased SGD without requiring bounded iterates and confirms function-value convergence rates in both local and global settings.

Theoretical Contributions

1. Convergence Analysis: The paper presents Theorem 1.5, which assures weak and function-value convergence for biased SGD without imposing restrictions like iterate boundedness. This means that the usual assumption that iterative trajectories remain bounded is unnecessary for convergence, thus broadening the applicability of the results.
2. Convergence Rates: Theorem 1.17 lays out convergence rates with high probability under specified learning rates, leveraging local and global Łojasiewicz frameworks. The results thus offer practical implications for selecting learning rates and understanding the influence of the Łojasiewicz exponent on convergence speed.
3. Complexity and Global Convergence: Corollaries 1.21 and 1.25 quantify computational complexity and present global convergence results, respectively. Notably, the latter shows that under suitable conditions, SGD trajectories converge almost surely to a stationary point of the function, providing guarantees even in the presence of unbiased noise.

Implications and Future Directions

The paper's findings suggest significant implications for training deep neural networks and solving nonconvex optimization problems. By relaxing common assumptions, the research enables a more comprehensive understanding of SGD dynamics, which could prompt fewer assumptions in applied settings. Future investigations may develop these results further by addressing confined trajectories or studying the dependencies of convergence properties on neural network architectures.

In conclusion, this work offers rigorous insights into the theoretical underpinnings of biased stochastic gradient descent. It invites continued exploration into scalable and efficient optimization strategies for increasingly complex and high-dimensional machine learning models, potentially influencing fields such as deep learning where nonconvex optimization is prevalent.