Optimal combination of multiple uncertainty scores for better discriminability

Determine an optimal strategy for combining multiple uncertainty estimation scores for large language models—such as pairwise concatenations of scores like CoE-C and verbalized confidence within the inter-score Truth Anchoring (TAC) framework—to maximize discriminability between correct and incorrect responses as measured by AUC.

Background

The paper introduces Truth Anchoring (TAC), a post-hoc calibration method that maps raw uncertainty proxy scores to truth-aligned probabilities of correctness. While TAC reliably improves calibration and often improves AUC, its ability to enhance discriminability is limited when the underlying proxy contains little information about correctness.

To address discriminability, the authors explore "inter-score anchoring," where two different uncertainty scores (e.g., an internal-state based score like CoE-C and an output-token based score like verbalized confidence) are combined as inputs to the mapper. Empirically, some pairwise combinations improve AUC, suggesting complementary information across scores. However, the authors explicitly note that the optimal way to combine scores to further improve discriminability remains unresolved.