- The paper surveys runtime safety monitoring techniques for DNN-based perception, demonstrating plug-and-play solutions for real-time error detection.
- It compares input, internal, and output monitoring methods to address generalization errors, out-of-distribution issues, and adversarial attacks.
- The paper emphasizes the promise of integrated monitoring frameworks in enhancing robust and reliable operation in safety-critical applications.
Runtime Safety Monitoring of Deep Neural Networks for Perception: A Survey
Introduction
The paper "Runtime Safety Monitoring of Deep Neural Networks for Perception: A Survey" (2511.05982) addresses the critical challenge of ensuring the safety and reliability of deep neural networks (DNNs) in perception systems, particularly in safety-critical applications like autonomous driving and robotics. The inherent vulnerabilities of DNNs to generalization errors, out-of-distribution (OOD) inputs, and adversarial attacks necessitate robust safety assurances. Traditional approaches such as exhaustive testing and retraining are often insufficient in dynamic real-world environments, thus motivating the need for runtime safety monitors (RSMs).
Runtime safety monitors (RSMs) operate parallel to DNNs during inference, enabling real-time detection of safety concerns without modifying the monitored network. By observing inputs, internal representations, and outputs, RSMs offer a non-intrusive, plug-and-play solution that complements traditional testing and certification processes, enhancing the reliability and trustworthiness of DNNs.
Previous Surveys
The field has seen several surveys focusing on the safety and reliability of DNNs, primarily during training or design phases. The paper by Huang et al. provides a detailed review of design-time safety assurance methods, including formal verification and coverage testing [huang2020asruvey]. Contrastingly, runtime methods are explored less frequently, highlighting the novelty of this survey.
Rahman et al.'s survey categorizes RSM approaches based on mobile robot perception into three groups: (1) past experiences, (2) inconsistencies during inference, and (3) confidence and uncertainty estimation [rahman2021runtime]. The current survey narrows its focus to methods that do not require training-phase data, emphasizing real-time detection during deployment.
Ferreira et al. introduce categorizations based on the integration of RSMs into systems, specifically distinguishing between external and internal methods [ferreira2023runtime]. This survey aligns with external monitoring approaches as they offer compatibility benefits.
Generalization Errors: These errors occur when DNNs fail to accurately predict outcomes for inputs within the training distribution, often due to overfitting or lack of diversity in training datasets [Jin2019QuantifyingTG].
Out-of-Distribution Errors: OOD errors arise when DNNs encounter inputs from different distributions than those seen during training, leading to overconfident but incorrect predictions [hendrycks2016baseline].
Adversarial Attacks: These attacks exploit vulnerabilities by introducing imperceptible noise patterns that lead to misclassification, posing significant security risks [Szegedy2013IntriguingPO].
Monitoring Approaches
This approach involves analyzing incoming data to preemptively detect anomalies before they are processed by the network. Techniques such as signal processing and reconstruction models are employed to identify irregularities in embeddings of input distributions.
Liu et al. focus on local robustness verification to distinguish adversarial from valid inputs based on robustness radii [Liu2023InputVF]. Asad et al. propose adversarial autoencoders for detecting OOD samples through reconstruction error analysis [asad2024beyond].
Monitoring Internal Representations
Internal monitoring methods observe neuron activations to detect irregularities during inference. Henzinger et al. utilize neuron activation patterns within hidden layers to capture OOD inputs, classifying them outside predefined activation bounds [Henzinger2019OutsideTB]. Yatbaz et al. leverage early-layer activation patterns in 3D object detection networks, successfully identifying missed detections in sparse LiDAR environments [Yatbaz2024RuntimeMO].
Monitoring Outputs
Simple yet effective, output monitoring uses prediction-level analysis like softmax entropy analysis and Monte Carlo dropout to estimate uncertainty. Kantaros et al. employ transformations of outputs to detect adversarial perturbations using statistical variance and divergence metrics [kantaros2021real].
Combined Approaches
Integration of multiple monitoring strategies helps address diverse safety concerns. Klingner et al. propose monitoring edge consistency between depth and segmentation outputs to detect adversarial perturbations in multi-task networks [klingner2022detecting]. Hacker et al. have developed a meta-model framework aggregating outputs from various monitors targeting different insufficiencies [hacker2023insufficiency].
Discussion
The surveyed runtime safety monitoring strategies underscore the complexity of ensuring safe DNN deployment across critical applications. No single method comprehensively addresses all vulnerabilities. Input monitoring captures explicit anomalies but falls short in detecting subtler issues, while internal monitoring provides deeper insights at the cost of higher computational demand. Output monitoring, although simple, suffers from limitations inherent to DNN overconfidence.
Integrated frameworks combining multiple monitoring strategies offer promising avenues for enhancing robustness in DNN-based systems. As adoption of DNNs in safety-critical domains escalates, advancements in runtime safety monitoring will be pivotal in maintaining operational safety and reliability.
Conclusion
Runtime safety monitoring embodies a crucial aspect of ensuring DNN reliability in perception tasks, particularly within safety-critical environments. The breadth of methods detailed in the survey, ranging from input to output monitoring, highlights their complementarities and respective strengths. Future research should focus on the adaptability, efficiency, and scalability of these frameworks, ensuring effective real-time deployment and safe operation.