Deep learning with Elastic Averaging SGD

Abstract: We study the problem of stochastic optimization for deep learning in the parallel computing environment under communication constraints. A new algorithm is proposed in this setting where the communication and coordination of work among concurrent processes (local workers), is based on an elastic force which links the parameters they compute with a center variable stored by the parameter server (master). The algorithm enables the local workers to perform more exploration, i.e. the algorithm allows the local variables to fluctuate further from the center variable by reducing the amount of communication between local workers and the master. We empirically demonstrate that in the deep learning setting, due to the existence of many local optima, allowing more exploration can lead to the improved performance. We propose synchronous and asynchronous variants of the new algorithm. We provide the stability analysis of the asynchronous variant in the round-robin scheme and compare it with the more common parallelized method ADMM. We show that the stability of EASGD is guaranteed when a simple stability condition is satisfied, which is not the case for ADMM. We additionally propose the momentum-based version of our algorithm that can be applied in both synchronous and asynchronous settings. Asynchronous variant of the algorithm is applied to train convolutional neural networks for image classification on the CIFAR and ImageNet datasets. Experiments demonstrate that the new algorithm accelerates the training of deep architectures compared to DOWNPOUR and other common baseline approaches and furthermore is very communication efficient.

• The paper's main contribution is introducing Elastic Averaging SGD, which enhances deep learning by allowing for greater exploration through an elastic force mechanism.
• It details both synchronous and asynchronous updates, with the momentum-based variant (EAMSGD) further accelerating convergence and reducing communication bottlenecks.
• Experimental results on benchmarks like CIFAR-10 and ImageNet validate its scalability and improved performance over traditional distributed SGD methods.

Deep Learning with Elastic Averaging SGD

The paper presents Elastic Averaging SGD (EASGD), a novel approach to enhance stochastic gradient descent (SGD) for deep learning models in parallel computing environments with communication constraints. The authors propose this algorithm to address the challenge of efficiently parallelizing the training of large-scale models that rely on SGD, such as convolutional neural networks (CNNs).

Algorithm Overview

EASGD introduces the concept of an "elastic force" that links the parameters computed by local workers with a central parameter maintained by a parameter server. This elastic mechanism allows local parameters to deviate more from the central variable than traditional methods, encouraging exploration in the parameter space. This exploration is essential in deep learning due to the presence of numerous local optima.

Two variants of EASGD are discussed: synchronous and asynchronous. The asynchronous version demonstrates particular promise by enabling workers to compute independently, reducing the communication overhead. A momentum-based variant, EAMSGD, is also proposed, which incorporates Nesterov's momentum to enhance convergence speed.

Technical Contributions

1. Stability Analysis: The stability of the asynchronous EASGD is analyzed, specifically under a round-robin scheme, and it is shown that the stability is ensured if certain conditions are met. The paper highlights that these conditions are not equally satisfied by methods like ADMM, showcasing EASGD's robustness.
2. Communication Efficiency: Through experiments on CIFAR and ImageNet datasets, EASGD and its momentum variant significantly reduce communication overhead while accelerating training time compared to established baseline methods like DOWNPOUR.
3. Exploration vs. Exploitation: The algorithm's allowance for greater parameter fluctuation enables better navigation of local optima, which theoretically and empirically results in improved performance on complex datasets.

Experimental Results

EASGD and EAMSGD are empirically demonstrated to outperform DOWNPOUR and several SGD variants on benchmark datasets such as CIFAR-10 and ImageNet. The experiments underline the algorithms' ability to leverage multiple GPU processors effectively, with EAMSGD particularly excelling due to its enhanced exploration capacity, leading to better test accuracy with fewer computational resources.

Implications and Future Directions

The implications of this work are significant for large-scale deep learning, where parallelization and communication bottlenecks are critical hurdles. EASGD's framework offers a pathway to more scalable model training, potentially influencing both theoretical insights and practical implementations in distributed machine learning systems.

Future research may focus on further optimizing the communication-accuracy trade-off, exploring adaptive adjustment of the elastic force, and extending the framework to other machine learning paradigms beyond CNNs. The exploration of additional theoretical properties, such as optimal convergence rates and complexity bounds in non-convex scenarios, would also strengthen the understanding and application of elastic averaging mechanisms in stochastic optimization.

In conclusion, the paper's introduction of EASGD reflects a meaningful advancement in parallelizing deep learning optimizations, emphasizing stability, efficiency, and the crucial balance between local exploration and global convergence.