MetaDetect: Uncertainty Quantification and Prediction Quality Estimates for Object Detection

Abstract: In object detection with deep neural networks, the box-wise objectness score tends to be overconfident, sometimes even indicating high confidence in presence of inaccurate predictions. Hence, the reliability of the prediction and therefore reliable uncertainties are of highest interest. In this work, we present a post processing method that for any given neural network provides predictive uncertainty estimates and quality estimates. These estimates are learned by a post processing model that receives as input a hand-crafted set of transparent metrics in form of a structured dataset. Therefrom, we learn two tasks for predicted bounding boxes. We discriminate between true positives ($\mathit{IoU}\geq0.5$) and false positives ($\mathit{IoU} < 0.5$) which we term meta classification, and we predict $\mathit{IoU}$ values directly which we term meta regression. The probabilities of the meta classification model aim at learning the probabilities of success and failure and therefore provide a modelled predictive uncertainty estimate. On the other hand, meta regression gives rise to a quality estimate. In numerical experiments, we use the publicly available YOLOv3 network and the Faster-RCNN network and evaluate meta classification and regression performance on the Kitti, Pascal VOC and COCO datasets. We demonstrate that our metrics are indeed well correlated with the $\mathit{IoU}$. For meta classification we obtain classification accuracies of up to 98.92% and AUROCs of up to 99.93%. For meta regression we obtain an $R^2$ value of up to 91.78%. These results yield significant improvements compared to other network's objectness score and other baseline approaches. Therefore, we obtain more reliable uncertainty and quality estimates which is particularly interesting in the absence of ground truth.

• The paper introduces a novel post-processing framework that employs meta classification and regression to quantify uncertainty and predict IoU values.
• It leverages interpretable metrics to significantly improve object detection performance, achieving up to 98.92% accuracy and 91.78% R² in regression.
• The method is network-agnostic, enhancing reliability in safety-critical applications such as autonomous systems.

MetaDetect: Uncertainty Quantification and Prediction Quality Estimates for Object Detection

Introduction

"MetaDetect: Uncertainty Quantification and Prediction Quality Estimates for Object Detection" (2010.01695) addresses the issue of overconfidence in box-wise objectness scores predicted by deep neural networks for object detection tasks. This research introduces a post-processing method that provides predictive uncertainty and quality estimates independent of network confidence scores. By leveraging a structured dataset comprising hand-crafted transparent metrics, the method learns to differentiate true positives (with an Intersection over Union, IoU, $\geq 0.5$) from false positives ($IoU < 0.5$) through meta classification and predicts IoU values directly via meta regression. The method was evaluated using YOLOv3 and Faster-RCNN networks on various datasets, including Kitti, Pascal VOC, and COCO, demonstrating superior correlation with IoU compared to existing approaches.

Methodology

The method proposed consists of a post-processing framework that is applicable to any neural network without altering its architecture or training protocol. The framework relies on a set of interpretable metrics derived from network output. These metrics consider various aspects such as candidate box quantity before NMS, score distribution, class probabilities, box dimension and aspect ratio, among others. The framework facilitates two meta tasks: meta classification to distinguish between true and false positives and meta regression to predict IoU values.

Figure 1: Examples of predicted bounding boxes with objectness score / true $IoU$ / predicted $IoU$. The predictions, and therefore the data to train and evaluate gradient boosting, are made with the YOLOv3 network, the Kitti dataset and a score threshold $t=0.01$.

Experimental Results

Extensive numerical experiments were performed across three datasets—Kitti, Pascal VOC, and COCO—using YOLOv3 and Faster-RCNN networks. Results indicated that meta classification achieved classification accuracies up to 98.92% and AUROCs as high as 99.93%, while meta regression yielded an $R^2$ of up to 91.78%.

Figure 2: Kitti + YOLOv3

The demonstrated results significantly improve upon standard network objectness scores and other baseline approaches, suggesting highly reliable uncertainty and quality estimates, especially in scenarios lacking ground truth data. The correlation between the proposed metrics and the actual IoU was strong, particularly with score-related metrics exhibiting coefficients exceeding 0.8.

Figure 3: Box-wise scatter plot of true IoU and predicted IoU values for the Kitti dataset, the YOLOv3 network and a score threshold $t=0.01. The predicted IoU values are generated with gradient boosting.

Implications and Future Work

The implications of MetaDetect are considerable for the field of object detection, especially in contexts demanding high reliability such as autonomous systems and safety-critical applications. Post-processing techniques that provide predictive uncertainty estimates can significantly enhance model interpretability and performance in practical deployments. However, the absence of improvement upon further incorporating Monte-Carlo dropout metrics suggests the necessity for more stochastic inference mechanisms to bolster uncertainty quantification.

Figure 4: Examples of predicted bounding boxes with a true $IoU=0$ but with high meta classification probabilities, indicating the presence of an object.

In future work, integrating this methodology with active learning pipelines and leveraging improved quality estimates to address data quality errors in labeling is recommended. Moreover, developing mechanisms to enhance the stochastic behavior of predictions may further advance the reliability of uncertainty estimates.

Conclusion

MetaDetect significantly enhances object detection reliability by offering robust predictive uncertainty and quality estimates independent of traditional confidence measures. Evaluated through comprehensive experiments, the method outperforms baseline approaches, evidencing its potential for real-world applications requiring high accuracy and dependability. Future exploration into active learning integration and stochastic inference techniques remains promising for maximizing its practical utility.