AI can evolve without labels: self-evolving vision transformer for chest X-ray diagnosis through knowledge distillation

Abstract: Although deep learning-based computer-aided diagnosis systems have recently achieved expert-level performance, developing a robust deep learning model requires large, high-quality data with manual annotation, which is expensive to obtain. This situation poses the problem that the chest x-rays collected annually in hospitals cannot be used due to the lack of manual labeling by experts, especially in deprived areas. To address this, here we present a novel deep learning framework that uses knowledge distillation through self-supervised learning and self-training, which shows that the performance of the original model trained with a small number of labels can be gradually improved with more unlabeled data. Experimental results show that the proposed framework maintains impressive robustness against a real-world environment and has general applicability to several diagnostic tasks such as tuberculosis, pneumothorax, and COVID-19. Notably, we demonstrated that our model performs even better than those trained with the same amount of labeled data. The proposed framework has a great potential for medical imaging, where plenty of data is accumulated every year, but ground truth annotations are expensive to obtain.

• The paper introduces a self-evolving framework that uses self-supervised learning and knowledge distillation to enhance diagnostic performance with unlabeled chest X-ray data.
• The method leverages Vision Transformers to iteratively improve semantic feature extraction and provide robust attention localization.
• Experimental results demonstrate that the proposed framework outperforms traditional models by reducing overfitting and enabling effective deployment in resource-limited settings.

The paper "AI can evolve without labels: self-evolving vision transformer for chest X-ray diagnosis through knowledge distillation" introduces a novel deep learning framework designed to improve the diagnostic performance of AI models used in analyzing chest X-ray images, even in scenarios where labeled data is scarce. This framework leverages a combination of self-supervised learning and knowledge distillation methods to enhance model robustness and performance using large amounts of unlabeled data. Here is a detailed summary of the key aspects covered in the paper:

Problem Statement

The paper addresses the challenge in medical imaging where vast amounts of data, such as chest X-rays, are available, but expert-labeled annotations are scarce, especially in resource-limited settings. Traditional deep learning models rely heavily on manual annotations, which are expensive and time-consuming to obtain. The goal is to develop a method allowing AI models to improve their performance using unlabeled data.

Proposed Framework

The authors propose a self-evolving framework named "DISTL" (Distillation for Self-supervised and Self-training Learning) that uses Vision Transformers (ViT) to gradually improve diagnostic performance with increasing amounts of unlabeled data. The framework consists of two main components:

1. Self-Supervised Learning: This component aims to learn task-agnostic semantic features from images, improving the model’s understanding of image content without relying on labels.
2. Self-Training with Knowledge Distillation: This involves training a student model to match the predictions of a teacher model. The student model is exposed to noisy or augmented versions of the images while the teacher model processes the clean versions.

Methodology

• Knowledge Distillation: The framework distills knowledge from a teacher model to a student model to refine predictions. The teacher model is gradually co-distilled from an updated student model, leveraging increased unlabeled data over time.
• Vision Transformer Utilization: By employing ViTs, which have a strong self-attention mechanism, the framework improves object understanding and attention localization without explicit supervision.
• Model Evolution: The framework allows the AI model to self-evolve over time as more unlabeled data become available, akin to iterative learning in humans, enhancing both semantic feature understanding and task-specific diagnostic capabilities.

Experimental Evaluation

The framework was evaluated across multiple diagnostic tasks, including tuberculosis, pneumothorax, and COVID-19 diagnosis using chest X-rays. Key findings include:

• The AI models trained with the proposed framework showed steadily improving performance as more unlabeled data were introduced, outperforming models trained only with labeled data.
• The ViT-based framework demonstrated superior performance and robustness against data from various clinical settings and devices.
• The experiments showed that the DISTL method could mitigate overfitting issues commonly encountered in models trained with limited labeled data.

Results and Implications

The self-evolving framework successfully demonstrated the ability to improve diagnostic accuracy in medical imaging tasks without the need for extensive labeled datasets. It provides a practical solution for implementing AI models in resource-constrained environments where labeled data are scarce. Additionally, the method's robustness to unlabeled data noise and its applicability to various tasks highlight its potential for widespread use in medical imaging.

In summary, the authors present a promising approach to harnessing the growth of unlabeled medical imaging data, potentially transforming how AI models are developed and deployed in clinical settings, reinforcing the autonomous and adaptive capabilities of AI in healthcare.