DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs

Abstract: As Deep Learning (DL) systems are widely deployed for mission-critical applications, debugging such systems becomes essential. Most existing works identify and repair suspicious neurons on the trained Deep Neural Network (DNN), which, unfortunately, might be a detour. Specifically, several existing studies have reported that many unsatisfactory behaviors are actually originated from the faults residing in DL programs. Besides, locating faulty neurons is not actionable for developers, while locating the faulty statements in DL programs can provide developers with more useful information for debugging. Though a few recent studies were proposed to pinpoint the faulty statements in DL programs or the training settings (e.g. too large learning rate), they were mainly designed based on predefined rules, leading to many false alarms or false negatives, especially when the faults are beyond their capabilities. In view of these limitations, in this paper, we proposed DeepFD, a learning-based fault diagnosis and localization framework which maps the fault localization task to a learning problem. In particular, it infers the suspicious fault types via monitoring the runtime features extracted during DNN model training and then locates the diagnosed faults in DL programs. It overcomes the limitations by identifying the root causes of faults in DL programs instead of neurons and diagnosing the faults by a learning approach instead of a set of hard-coded rules. The evaluation exhibits the potential of DeepFD. It correctly diagnoses 52% faulty DL programs, compared with around half (27%) achieved by the best state-of-the-art works. Besides, for fault localization, DeepFD also outperforms the existing works, correctly locating 42% faulty programs, which almost doubles the best result (23%) achieved by the existing works.

• The paper introduces DeepFD, a novel framework that automates fault diagnosis in DL programs by mapping the task into a learning problem.
• The paper extracts dynamic runtime features such as loss, gradients, and accuracy to reveal correlations with fault types and enable precise fault localization.
• The paper demonstrates that DeepFD, employing multiple machine learning models, significantly improves diagnosis accuracy by doubling correct fault localization rates compared to existing methods.

DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs

Introduction

As Deep Learning (DL) systems are increasingly utilized in mission-critical applications, ensuring their reliability becomes paramount. Faulty behaviors in such systems often originate from deficiencies within DL programs rather than trained models, presenting challenges in debugging. Conventional approaches that focus on neurons are less actionable for developers. In response, DeepFD is introduced as an innovative framework that redefines fault diagnosis and localization in DL programs through a learning-based approach.

Overview of DeepFD

DeepFD addresses the limitations of hard-coded debugging rules by mapping the fault localization task to a learning problem. By monitoring runtime features during DNN model training, the framework identifies suspicious fault types and locates the corresponding faulty code in DL programs.

Figure 1: Overview of DeepFD.

DeepFD's approach emphasizes understanding the root causes of faults, focusing on program code statements rather than neurons. By leveraging dynamic feature extraction, the system learns nuanced patterns associated with common fault types.

Diagnostic Feature Extraction

DeepFD collects a rich set of runtime data, including loss, gradients, and validation accuracy, from training phases. These data are then processed using statistical methods to derive diagnostic features that characterize the training process.

Figure 2: Correlation Between Types of Faults and Features.

The diagnostic features reveal correlations between training behaviors and specific faults, such as oscillations in loss indicating improper learning rates. This extraction forms the basis for the framework’s learning-driven fault diagnosis.

Diagnosis Model Construction

DeepFD employs multiple machine learning models to classify fault types based on extracted features. Models constructed using algorithms like K-Nearest Neighbors, Decision Tree, and Random Forest are trained with a benchmark dataset composed of fault-seeded DL programs.

Figure 3: The Workflow of Diagnosis Models Construction.

By translating fault localization into a classification problem, DeepFD can dynamically infer fault types from runtime training patterns, offering actionable insights for developers to remedy issues.

Evaluation and Benchmark

The DeepFD framework was evaluated using a comprehensive benchmark of DL programs containing diverse real-world faults. It demonstrated an improvement in diagnosis accuracy compared to existing state-of-the-art methods, effectively doubling the correct diagnosis and localization rates.

Figure 4: The Statistics of Benchmark.

DeepFD’s learning-based methodology exhibits significant potential in diagnosing and locating various types of faults, paving the way for robust DL systems.

Conclusion

DeepFD represents a pivotal advancement in fault diagnosis and localization within DL programs. Through its learning-based approach, it successfully overcomes the drawbacks of predefined rules, enabling developers to identify and fix root causes of faults efficiently. This framework not only enhances debugging accuracy but also serves as a foundation for future explorations in automated DL program repair.