Proper Scoring Rules and Bregman Divergences

Abstract: We revisit the mathematical foundations of proper scoring rules (PSRs) and Bregman divergences and present their characteristic properties in a unified theoretical framework. In many situations it is preferable not to generate a PSR directly from its convex entropy on the unit simplex but instead by the sublinear extension of the entropy to the positive orthant. This gives the scoring rule simply as a subgradient of the extended entropy, allowing for a more elegant theory. The other convex extensions of the entropy generate affine extensions of the scoring rule and induce the class of functional Bregman divergences. We discuss the geometric nature of the relationship between PSRs and Bregman divergences and extend and unify existing partial results. We also approach the topic of differentiability of entropy functions. Not all entropies of interest possess functional derivatives, but they do all have directional derivatives in almost every direction. Relying on the notion of quasi-interior of a convex set to quantify the latter property, we formalise under what conditions a PSR may be considered to be uniquely determined from its entropy.