Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration

Abstract: Adam is one of the most influential adaptive stochastic algorithms for training deep neural networks, which has been pointed out to be divergent even in the simple convex setting via a few simple counterexamples. Many attempts, such as decreasing an adaptive learning rate, adopting a big batch size, incorporating a temporal decorrelation technique, seeking an analogous surrogate, \textit{etc.}, have been tried to promote Adam-type algorithms to converge. In contrast with existing approaches, we introduce an alternative easy-to-check sufficient condition, which merely depends on the parameters of the base learning rate and combinations of historical second-order moments, to guarantee the global convergence of generic Adam for solving large-scale non-convex stochastic optimization. This observation, coupled with this sufficient condition, gives much deeper interpretations on the divergence of Adam. On the other hand, in practice, mini-Adam and distributed-Adam are widely used without any theoretical guarantee. We further give an analysis on how the batch size or the number of nodes in the distributed system affects the convergence of Adam, which theoretically shows that mini-batch and distributed Adam can be linearly accelerated by using a larger mini-batch size or a larger number of nodes.At last, we apply the generic Adam and mini-batch Adam with the sufficient condition for solving the counterexample and training several neural networks on various real-world datasets. Experimental results are exactly in accord with our theoretical analysis.

• The paper reveals a sufficient condition that ensures global convergence of Adam in non-convex stochastic problems.
• It extends the analysis to mini-batch and distributed settings, showing linear acceleration with larger batch sizes and more nodes.
• Experimental validation on datasets like MNIST and CIFAR-100 confirms improved stability and faster convergence in practical scenarios.

Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration

Introduction

The paper focuses on the convergence behavior of Adam, a prevalent optimizer in deep learning, which has been criticized for potential divergence in certain convex scenarios. A novel sufficient condition is proposed to ensure the global convergence of Adam in non-convex stochastic settings. The theoretical analysis is extended to mini-batch and distributed Adam, demonstrating linear speed-up with larger batch sizes or distributed nodes.

Convergence Analysis of Generic Adam

The authors detail a sufficient condition to guarantee convergence, specifically designed to be easily verifiable and independent of complex modifications. This condition depends solely on the base learning rate and historical second-order moments:

1. Parameters $\beta_t$, $\theta_t$, and $\alpha_t$ are selected to maintain certain mathematical properties.
2. The convergence rate is impacted by the balance between base learning rate and second-order moment combinations.

Practical Extensions: Mini-Batch and Distributed Adam

The analysis is extended to practical scenarios where Adam optimizers are used on large-scale data:

• Mini-Batch Adam: By employing larger mini-batches, the convergence speed shows a linear acceleration with respect to batch size.
• Distributed Adam: Implementing Adam in a parameter-server model shows potential linear speed-up proportional to distributed nodes.

Such extensions bridge theoretical findings with practical implementations, reflecting the real-world improvement in optimization tasks.

Experimental Validation

Experiments conducted involve synthetic counterexamples and real-world data sets (e.g., MNIST, CIFAR-100) for neural networks like LeNet and ResNet. These validate the proposed sufficient condition, illustrating practical benefits such as convergence acceleration and stability improvements. Key insights include:

• Catalyst for faster convergence with increasing parameter $r$.
• Practical performance gains through larger mini-batch sizes in mini-batch Adam.

Figure 1: The above figures showcase function values with varying $r$ and $s$ values, providing a visual comparison of Generic Adam’s performance.

Conclusion

The study provides a theoretical foundation and practical guidelines for employing Adam in non-convex optimization tasks, emphasizing the necessity of parameter tuning for convergence. Through rigorous experiments, the authors demonstrate the practical feasibility and performance gains achievable via mini-batch and distributed approaches. This extension of Adam's theoretical understanding supports its robust application in large-scale machine learning problems, highlighting speed-up and stability in real-world scenarios.

In essence, this work lays the groundwork for optimizing neural networks with enhanced efficiency and reliability, catering to the ever-increasing demands of large-scale AI deployments.