Weighted Difference Approximation of Value Functions for Slow-Discounting Markov Decision Processes

Abstract: Processes (MDPs) often require frequent decision making, that is, taking an action every microsecond, second, or minute. Infinite horizon discount reward formulation is still relevant for a large portion of these applications, because actual time span of these problems can be months or years, during which discounting factors due to e.g. interest rates are of practical concern. In this paper, we show that, for such MDPs with discount rate $\alpha$ close to $1$, under a common ergodicity assumption, a weighted difference between two successive value function estimates obtained from the classical value iteration (VI) is a better approximation than the value function obtained directly from VI. Rigorous error bounds are established which in turn show that the approximation converges to the actual value function in a rate $(\alpha \beta)^k$ with $\beta<1$. This indicates a geometric convergence even if discount factor $\alpha \to 1$. Furthermore, we explicitly link the convergence speed to the system behaviors of the MDP using the notion of $\epsilon-$mixing time and extend our result to Q-functions. Numerical experiments are conducted to demonstrate the convergence properties of the proposed approximation scheme.