Convergence of A3C’s asynchronous updates

Prove convergence of the Asynchronous Advantage Actor–Critic (A3C) algorithm’s lock-free asynchronous parameter updates in the parallel-worker setting, or derive rigorous conditions under which these asynchronous updates converge.

Background

A3C achieves parallel, lock-free updates by running multiple workers that asynchronously update shared parameters, removing the need for experience replay and reducing correlation in gradient estimates. Despite strong empirical performance, the theoretical understanding of convergence for these asynchronous updates is incomplete.

Existing analyses of asynchronous stochastic approximation focus on classical asynchrony within a single trajectory and do not address the parallel-worker gradient regime used by A3C, leaving a gap in rigorous convergence guarantees.