Tighter Confidence Intervals for Rating Systems

Abstract: Rating systems are ubiquitous, with applications ranging from product recommendation to teaching evaluations. Confidence intervals for functionals of rating data such as empirical means or quantiles are critical to decision-making in various applications including recommendation/ranking algorithms. Confidence intervals derived from standard Hoeffding and Bernstein bounds can be quite loose, especially in small sample regimes, since these bounds do not exploit the geometric structure of the probability simplex. We propose a new approach to deriving confidence intervals that are tailored to the geometry associated with multi-star/value rating systems using a combination of techniques from information theory, including Kullback-Leibler, Sanov, and Csisz{\'a}r inequalities. The new confidence intervals are almost always as good or better than all standard methods and are significantly tighter in many situations. The standard bounds can require several times more samples than our new bounds to achieve specified confidence interval widths.