Tighter Analysis for Decentralized Stochastic Gradient Method: Impact of Data Homogeneity

Abstract: This paper studies the effect of data homogeneity on multi-agent stochastic optimization. We consider the decentralized stochastic gradient (DSGD) algorithm and perform a refined convergence analysis. Our analysis is explicit on the similarity between Hessian matrices of local objective functions which captures the degree of data homogeneity. We illustrate the impact of our analysis through studying the transient time, defined as the minimum number of iterations required for a distributed algorithm to achieve comparable performance as its centralized counterpart. When the local objective functions have similar Hessian, the transient time of DSGD can be as small as ${\cal O}(n^{2/3}/\rho^{8/3})$ for smooth (possibly non-convex) objective functions, ${\cal O}(\sqrt{n}/\rho)$ for strongly convex objective functions, where $n$ is the number of agents and $\rho$ is the spectral gap of graph. These findings provide a theoretical justification for the empirical success of DSGD. Our analysis relies on a novel observation with higher-order Taylor approximation for gradient maps that can be of independent interest. Numerical simulations validate our findings.