Linear Programming for Decision Processes with Partial Information

Abstract: Markov Decision Processes (MDPs) are stochastic optimization problems that model situations where a decision maker controls a system based on its state. Partially observed Markov decision processes (POMDPs) are generalizations of MDPs where the decision maker has only partial information on the state of the system. Decomposable POMDPs are specific cases of POMDPs that enable one to model systems with several components. Such problems naturally model a wide range of applications such as predictive maintenance. Finding an optimal policy for a POMDP is PSPACE-hard and practically challenging. We introduce a mixed integer linear programming (MILP) formulation for POMDPs restricted to the policies that only depend on the current observation, as well as valid inequalities that are based on a probabilistic interpretation of the dependence between variables. The linear relaxation provides a good bound for the usual POMDPs where the policies depend on the full history of observations and actions. Solving decomposable POMDPs is especially challenging due to the curse of dimensionality. Leveraging our MILP formulation for POMDPs, we introduce a linear program based on ``fluid formulation'' for decomposable POMDPs, that provides both a bound on the optimal value and a practically efficient heuristic to find a good policy. Numerical experiments show the efficiency of our approaches to POMDPs and decomposable POMDPs.