Computationally Efficient Methods for Solving Discrete-time Dynamic models with Continuous Actions

Abstract: This study investigates computationally efficient algorithms for solving discrete-time infinite-horizon single-agent/multi-agent dynamic models with continuous actions. It shows that we can easily reduce the computational costs by slightly changing basic algorithms using value functions, such as the Value Function Iteration (VFI) and the Policy Iteration (PI). The PI method with a Krylov iterative method (GMRES), which can be easily implemented using built-in packages, works much better than VFI-based algorithms even when considering continuous state models. Concerning the VFI algorithm, we can largely speed up the convergence by introducing acceleration methods of fixed-point iterations. The current study also proposes the VF-PGI-Spectral (Value Function-Policy Gradient Iteration Spectral) algorithm, which is a slight modification of the VFI. It shows numerical results where the VF-PGI-Spectral performs much better than the VFI- and PI-based algorithms especially in multi-agent dynamic games. Finally, it shows that using relative value functions further reduces the computational cost of these methods.