$N$-Timescale Stochastic Approximation: Stability and Convergence

Abstract: This paper presents the first sufficient conditions that guarantee the stability and almost sure convergence of $N$-timescale stochastic approximation (SA) iterates for any $N\geq1$. It extends the existing results on One-timescale and Two-timescale SA iterates to general $N$-timescale stochastic recursions using the ordinary differential equation (ODE) method. As an application of our results, we study SA algorithms with an added heavy ball momentum term in the context of Gradient Temporal Difference (GTD) algorithms. We show that, when the momentum parameters are chosen in a certain way, the schemes are stable and convergent to the same solution using our proposed results.