Threshold-free Evaluation of Medical Tests for Classification and Prediction: Average Precision versus Area Under the ROC Curve

Abstract: When evaluating medical tests or biomarkers for disease classification, the area under the receiver-operating characteristic (ROC) curve is a widely used performance metric that does not require us to commit to a specific decision threshold. For the same type of evaluations, a different metric known as the average precision (AP) is used much more widely in the information retrieval literature. We study both metrics in some depths in order to elucidate their difference and relationship. More specifically, we explain mathematically why the AP may be more appropriate if the earlier part of the ROC curve is of interest. We also address practical matters, deriving an expression for the asymptotic variance of the AP, as well as providing real-world examples concerning the evaluation of protein biomarkers for prostate cancer and the assessment of digital versus film mammography for breast cancer screening.