Empirical evaluation of uncertainty quantification methods in supervised learning

Develop empirical evaluation methodologies for assessing methods that quantify aleatoric, epistemic, and total predictive uncertainty in supervised learning, despite the absence of ground-truth uncertainty labels in typical datasets.

Background

The paper notes that, unlike target variables, datasets typically do not provide ground-truth uncertainty labels, which makes it difficult to directly assess the correctness of predicted aleatoric, epistemic, or total uncertainty. Consequently, most current evaluations are indirect, for example by measuring the usefulness of uncertainty estimates for improved prediction and decision-making (e.g., through set-valued prediction).

The authors mention accuracy–rejection curves as one such indirect assessment, where a classifier’s accuracy is plotted against the fraction of abstentions. However, they highlight that a principled, general methodology for empirical evaluation of uncertainty quantification methods remains to be established.