Optimal Decision Rules for Simple Hypothesis Testing under General Criterion Involving Error Probabilities

Abstract: The problem of simple $M-$ary hypothesis testing under a generic performance criterion that depends on arbitrary functions of error probabilities is considered. Using results from convex analysis, it is proved that an optimal decision rule can be characterized as a randomization among at most two deterministic decision rules, of the form reminiscent to Bayes rule, if the boundary points corresponding to each rule have zero probability under each hypothesis. Otherwise, a randomization among at most $M(M-1)+1$ deterministic decision rules is sufficient. The form of the deterministic decision rules are explicitly specified. Likelihood ratios are shown to be sufficient statistics. Classical performance measures including Bayesian, minimax, Neyman-Pearson, generalized Neyman-Pearson, restricted Bayesian, and prospect theory based approaches are all covered under the proposed formulation. A numerical example is presented for prospect theory based binary hypothesis testing.