Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control

Abstract: Reinforcement learning (RL) is currently one of the most popular methods, with breakthrough results in a variety of fields. The framework relies on the concept of Markov decision process (MDP), which corresponds to a discrete time optimal control problem. In the RL literature, such problems are usually formulated with mixed policies, from which a random action is sampled at each time step. Recently, the optimal control community has studied continuous-time versions of RL algorithms, replacing MDPs with mixed policies by continuous time stochastic processes with relaxed controls. In this work, we rigorously connect the two problems: we prove the strong convergence of the former towards the latter when the time discretization goes to $0$.