Absorbing Markov Decision Processes: Geometric Properties and Sufficiency of Finite Mixtures of Deterministic Policies

Abstract: In this paper we investigate several geometric properties of the set of occupancy measures. In particular, we analyse the structure of the faces generated by a given occupancy measure, together with their relative algebraic interior. We also determine the affine hulls of these faces and describe the associated parallel linear subspaces. It is shown that these structures can be fully characterised in terms of the parameters that define the underlying Markov decision process (MDP). Moreover, we establish that the class of finite mixtures of deterministic stationary policies constitutes a sufficient class of policies for uniformly absorbing MDPs with a measurable state space and multiple criteria. We also provide a characterisation of the minimal order required for a finite mixture of deterministic stationary policies to represent the performance vector of an arbitrary policy.