Contextual Speech Extraction: Leveraging Textual History as an Implicit Cue for Target Speech Extraction

Abstract: In this paper, we investigate a novel approach for Target Speech Extraction (TSE), which relies solely on textual context to extract the target speech. We refer to this task as Contextual Speech Extraction (CSE). Unlike traditional TSE methods that rely on pre-recorded enrollment utterances, video of the target speaker's face, spatial information, or other explicit cues to identify the target stream, our proposed method requires only a few turns of previous dialogue (or monologue) history. This approach is naturally feasible in mobile messaging environments where voice recordings are typically preceded by textual dialogue that can be leveraged implicitly. We present three CSE models and analyze their performances on three datasets. Through our experiments, we demonstrate that even when the model relies purely on dialogue history, it can achieve over 90 % accuracy in identifying the correct target stream with only two previous dialogue turns. Furthermore, we show that by leveraging both textual context and enrollment utterances as cues during training, we further enhance our model's flexibility and effectiveness, allowing us to use either cue during inference, or combine both for improved performance. Samples and code available on https://miraodasilva.github.io/cse-project-page .

Contextual Speech Extraction: Leveraging Textual History as an Implicit Cue for Target Speech Extraction

The paper presents a novel approach to Target Speech Extraction (TSE) termed Contextual Speech Extraction (CSE), which leverages textual history as an implicit cue, distinguishing itself from traditional methods requiring explicit cues. This study is motivated by the inherent constraints traditional TSE approaches impose by necessitating pre-recorded enrollment utterances, spatial information, or visual data. Such explicit cues pose challenges in practical scenarios, particularly in dynamic environments like mobile messaging where text-based dialogue histories are naturally present.

The authors elucidate three primary CSE models designed to utilize dialogue history without explicit cues: a cascaded model, a unified contextual separator (ContSep), and a contextual extractor (ContExt). In their experimental framework, the authors employ pre-trained language models to embed context and integrate this with a Sepformer-based architecture, demonstrating the efficacy of CSE models across diverse datasets, inclusive of dialogues and monologues. The study highlights that even with minimal context—such as two preceding dialogue turns—the models can achieve over 90% accuracy in identifying target streams. This performance underscores the potential of CSE in environments where obtaining traditional cues is cumbersome.

The cascaded approach combines Sepformer for speech separation, Whisper for transcription, and Llama for language modeling. However, this model, despite its baseline functionality, is prone to inefficiencies and error propagation, particularly in contexts with non-native speech patterns. Unified models such as ContSep and ContExt overcome these limitations by directly predicting the target speech using integrated modeling of the textual context, thus avoiding cumbersome intermediate transcription procedures. Specifically, ContExt is designed solely to extract the target stream, resulting in superior performance in several test scenarios over its counterparts.

Additionally, the introduction of Hybrid Contextual Extractor (H-ContExt) exemplifies an innovative confluence of textual context and enrollment utterance cues. This hybrid model confers flexibility, allowing either or both cues for extracting the target speech stream, demonstrating heightened adaptability and enhanced performance.

The study's quantitative results, particularly those obtained across three benchmark datasets—DailyTalk, SpokenWOZ, and TED-LIUM 3—offer empirical validation of CSE's potential. The utilization of context alone shows much promise, achieving competitive accuracy metrics compared to traditional TSE that relies on audio cues. Furthermore, the hybrid approach demonstrates superior flexibility without sacrificing performance, bridging the gap between the utilization of implicit versus explicit cues.

From a technical perspective, the integration of large language models (LLMs) with the Sepformer architecture is a critical innovation enabling the contextual embedding to directly influence the acoustic processing pathways. This aligns with the broader trend of incorporating LLMs into diverse audio processing tasks, supporting their role in enhancing speech extraction fidelity through comprehension of preceding context.

Implications of this research extend both practically and theoretically. Practically, CSE models promise to enhance real-time communication applications on platforms such as mobile messaging, where obtaining traditional cues may not be feasible. Theoretically, the validation of text as a viable cue challenges prevailing paradigms in speech processing and opens avenues for further exploration into hybrid architectures combining text and audio cues.

Future research may focus on refining CSE frameworks to further improve resilience in noisy environments and explore the integration of visual cues alongside textual ones, potentially broadening applicability in multimedia contexts. Additionally, optimizing the efficiency of LLM embeddings for real-time applications could address existing computational overheads.

In conclusion, this paper provides significant contributions to the field of speech processing by introducing and validating Contextual Speech Extraction. In leveraging text as an implicit cue, it sets a foundation for future innovations that prioritize flexibility and context-awareness in speech extraction technologies.