A Self-Training Approach for Whisper to Enhance Long Dysarthric Speech Recognition

Abstract: Dysarthric speech recognition (DSR) enhances the accessibility of smart devices for dysarthric speakers with limited mobility. Previously, DSR research was constrained by the fact that existing datasets typically consisted of isolated words, command phrases, and a limited number of sentences spoken by a few individuals. This constrained research to command-interaction systems and speaker adaptation. The Speech Accessibility Project (SAP) changed this by releasing a large and diverse English dysarthric dataset, leading to the SAP Challenge to build speaker- and text-independent DSR systems. We enhanced the Whisper model's performance on long dysarthric speech via a novel self-training method. This method increased training data and adapted the model to handle potentially incomplete speech segments encountered during inference. Our system achieved second place in both Word Error Rate and Semantic Score in the SAP Challenge.

• The paper introduces a novel self-training method where Whisper acts as a teacher to generate accurate pseudo-labels, improving long dysarthric speech segmentation.
• It employs an effective segmentation strategy with heuristic filtering to handle incomplete speech segments and significantly reduce word error rates.
• Experimental results using the SAP dataset and optimal beam decoding validate enhanced ASR performance and robust speaker-independent dysarthric speech recognition.

A Self-Training Approach for Whisper to Enhance Long Dysarthric Speech Recognition

Introduction

Automatic Speech Recognition (ASR) systems encounter challenges when dealing with dysarthric speech, characterized by impaired vocal control due to neurological conditions such as Parkinson’s disease and cerebral palsy. Traditional ASR models generally perform poorly on dysarthric speech, primarily because these models are trained on typical speech data, which lacks the diverse acoustic and prosodic variations present in dysarthric speech. Dysarthric speech recognition (DSR) has historically been constrained to command-level tasks due to the limited diversity of available datasets. With the emergence of the Speech Accessibility Project (SAP), a shift toward speaker-independent and text-independent DSR systems has been initiated, thereby promoting more advanced research in this domain.

The paper presents an innovative self-training approach designed to enhance Whisper's performance on long dysarthric speech. By leveraging the large and diverse SAP dataset, the researchers aimed to mitigate the data scarcity in DSR that typically hinders robust system development. This new methodology involves augmenting the training data, particularly catering to the handling of potentially incomplete speech segments during inference.

Figure 1: The overview of our self-training approach to segment long dysarthric speech (ST-SLDS).

Self-Training Optimization

The proposed self-training approach addresses the challenge of segmenting long dysarthric speech (ST-SLDS). Initially, short dysarthric speech samples are used to fine-tune the Whisper model as a teacher model. This teacher model then generates pseudo-labels for the unused long dysarthric speech through a segmentation-based inference approach. The segmentation occurs prior to inference, where long speech is divided into segments that fit within the model’s positional encoding limits without leading to incomplete predictions.

Filtered data, meeting specific heuristic criteria (e.g., zero WER, zero insertion, and deletion errors), are selected for further processing. This involves annotation via segmentation algorithms, where precise timestamps are used for segment division, and corresponding predictions are used as labels. Through iterative training, the segmented data is merged into the training pool, progressively advancing the model's robustness and adaptability.

Figure 2: Segmentation algorithm (a) and (b).

Experimental Results

Empirical evaluations were conducted using various advanced ASR models to form speaker-independent and text-independent DSR systems. The Whisper large-v3 model demonstrated superior performance, especially when employing an even segmentation (E-S) strategy during inference, which significantly improved the Word Error Rate (WER). The model was further refined through systematic experimentation with inference settings, including various segmentation and decoding strategies.

In evaluating different settings, a beam size of 10 was identified as optimal for decoding, with VAD-S strategy aiding better segmentation points compared to E-S. The proposed self-training method conspicuously enhanced performance by incrementally incorporating pseudo-labeled data filtered through a robust heuristic strategy. The inclusion of the SAP0430_processed dev partition in training brought additional improvement, although speaker complexity introduced challenges in aligning performance on new test datasets.

Figure 3: The duration distribution of the training data in SAP0430_processed.

Implications and Future Directions

The findings underscore the potential of self-training methodologies in addressing the gaps in dysarthric speech datasets, particularly for long-segment speech recognition. Enhancing Whisper models through iterative self-training presents a viable solution to the mismatch between training and inference scenarios. This approach not only improves recognition accuracy for long dysarthric speech but also demonstrates resilience in adapting to the varied acoustic features inherent in dysarthric speech.

Looking ahead, further refinement of segmentation strategies and heuristic filtering could yield additional performance enhancements. Continued exploration into adapting VAD tools specifically for dysarthric speech may also facilitate more accurate segmentation, thereby optimizing the model’s efficacy.

Conclusion

The award-winning system advocated in this paper represents a significant step toward refining dysarthric speech recognition systems. By systematically increasing training data through self-training and addressing training-inference mismatches, the proposed methodology exhibits substantial advancements in model performance. These insights and techniques contribute to the broader field of ASR, offering avenues for future research in adaptive speech recognition systems for speech disorders.