Continuous-time Markov Decision Processes with Finite-horizon Expected Total Cost Criteria

Abstract: This paper deals with the unconstrained and constrained cases for continuous-time Markov decision processes under the finite-horizon expected total cost criterion. The state space is denumerable and the transition and cost rates are allowed to be unbounded from above and from below. We give conditions for the existence of optimal policies in the class of all randomized history-dependent policies. For the unconstrained case, using the analogue of the forward Kolmogorov equation in the form of conditional expectation, we show that the finite-horizon optimal value function is the unique solution to the optimality equation and obtain the existence of an optimal deterministic Markov policy. For the constrained case, employing the technique of occupation measures, we first give an equivalent characterization of the occupation measures, and derive that for each occupation measure generated by a randomized history-dependent policy, there exists an occupation measure generated by a randomized Markov policy equal to it. Then using the compactness and convexity of the set of all occupation measures, we obtain the existence of a constrained-optimal randomized Markov policy. Moreover, the constrained optimization problem is reformulated as a linear program, and the strong duality between the linear program and its dual program is established. Finally, a controlled birth and death system is used to illustrate our main results.