Universal Speech Enhancement

• Universal Speech Enhancement is a unified approach that recovers intelligible speech from any degraded input, handling noise, reverberation, clipping, and other impairments.
• It leverages hybrid architectures that combine discriminative, generative, and self-supervised techniques to enhance fidelity, reduce artifacts, and maintain content integrity.
• Evaluation involves both intrusive and non-intrusive metrics like PESQ, SI-SDR, and MOS on multilingual and multi-condition datasets to ensure broad robustness.

Universal Speech Enhancement is the development and deployment of a single system capable of recovering high-quality, intelligible, and perceptually natural speech from any input audio signal subject to arbitrary and potentially unknown mixtures of distortions and recording conditions. Such distortions include, but are not limited to, additive noise, reverberation, clipping, codec and packet loss artifacts, bandwidth limitation, wind noise, and microphone anomalies. Universal systems must generalize across diverse languages, sampling rates (typically 8–48 kHz), utterance durations, speaker traits, and channel configurations, all without explicit information about the degradation type at inference (Saijo et al., 29 May 2025, Li et al., 20 Jan 2026, Zhang et al., 30 May 2025). This task represents a unification of previously siloed subproblems—denoising, dereverberation, dereverberation, bandwidth extension, and inpainting—within a single model architecture, trained on large-scale, multi-condition, and often multilingual datasets.

1. Scope and Problem Formulation

Universal speech enhancement (USE) generalizes classical enhancement by assuming a composite, unknown distortion process. The general signal model is: $y(t) = (s*h)(t) + n(t) + d(t)$ where $s(t)$ is clean speech, $h(t)$ is an unknown room impulse response, $n(t)$ is additive noise, and $d(t)$ encompasses all other non-linear or non-additive distortions (e.g., clipping, codec compression, bandwidth limitation, packet loss) (Li et al., 20 Jan 2026, Zhang et al., 30 May 2025).

Key requirements:

• Universality: One model is used for all distortions, channels, sampling rates, and languages.
• Distortion-agnosticism: The model must operate without knowledge of the distortion types at test time.
• Generalization: Robustness to unseen speakers, new noise types, and out-of-domain acoustic conditions.
• Joint metric optimization: Both intrusive (PESQ, ESTOI, SI-SDR) and non-intrusive (DNSMOS, NISQA), as well as downstream (ASR WER, speaker similarity) metrics must be improved.

These requirements preclude methods that require scenario-specific retraining, explicit distortion detection, or per-condition architectures (Zhang et al., 2023, Rong et al., 24 May 2025).

2. System Architectures and Methodologies

The evolution of universal SE models reflects a move from discriminative regression networks toward hybrid and generative systems integrating elements of deep learning, diffusion models, and discrete tokenization.

2.1 Discriminative Architectures

• Time-Frequency Grid Networks (TF-GridNet), Sub-band RNNs, Dual-path Transformers: These networks process complex STFT representations using multi-path processing for different axes (time, frequency, channel), incorporating layer normalization, attention, and sampling-rate-independent STFT frontends. Notable discriminative backbones include BSRNN and various forms of TF-GridNet (Li et al., 20 Jan 2026, Liu et al., 27 Jan 2026, Saijo et al., 29 May 2025, Zhang et al., 2023).
• Sampling-Frequency Independence (SFI): By fixing window/hop in milliseconds, STFT representations are consistent across different sampling rates, enabling single-model deployment over 8–48 kHz (Zhang et al., 2023, Liu et al., 27 Jan 2026).
• Memory tokens and channel-aggregation (TAC, CWS): Mechanisms to support arbitrary input length and microphone numbers (Zhang et al., 2023, Rong et al., 24 May 2025).

2.2 Generative and Hybrid Strategies

• Flow-Based and Diffusion Models: Generative approaches model the conditional distribution of clean speech given a degraded observation, using score-based diffusion (UNIVERSE++, PGUSE, SenSE), flow-matching (USEMamba, SenSE), or autoregressive discrete-token modeling (Scheibler et al., 2024, Zhang et al., 30 May 2025, Li et al., 29 Sep 2025, Chao et al., 27 May 2025).
• Discrete-Domain Enhancement: Predicting neural codec tokens of clean speech, leveraging residual vector quantization (RVQ), shifts the problem to classification, providing uniform handling of all distortion types (UDSE, FUSE stage 2) (Liu et al., 11 Oct 2025, Goswami et al., 1 Jun 2025).
• Hybrid Branches and Adaptive Fusion: Recent methods (PGUSE, FUSE, hybrid TF-GridNet+AR, TS-URGENet) combine discriminative and generative branches, fusing outputs via learned masks, blending, or late-stage restoration modules. Fusion is typically guided by a combination of perceptual (MOS-predictive), signal-level, and task-dependent losses (Liu et al., 27 Jan 2026, Goswami et al., 1 Jun 2025, Zhang et al., 30 May 2025).

2.3 Three-Stage and Modular Systems

• Filling–Separation–Restoration Pipelines: Multi-stage approaches such as TS-URGENet decompose enhancement sequentially: first filling in lost/noisy regions (e.g., packet loss), then local separation/denoising, and finally bandwidth/codec restoration (Rong et al., 24 May 2025).
• Token guidance and semantic priors: Using semantic-aware LLMs or AV-driven unit prediction (SenSE, ReVISE) infuses high-level content preservation and speaker-identity robustness into generative enhancement (Li et al., 29 Sep 2025, Hsu et al., 2022).
• Self-Supervised Foundational Encoders: Masked autoencoders employing a rich augmentation stack enable self-supervised universal encoders that can be frozen and fine-tuned for multiple tasks (MAE-USE) (Rajagopalan et al., 2 Feb 2026).

3. Training Objectives, Evaluation Metrics, and Benchmarks

Universal SE training combines multiple losses and is validated on composite, multilingual, and multi-distortion test sets.

3.1 Loss Functions

• Regression and Perceptual Losses: MSE, L1, multi-resolution STFT loss, and SI-SDR are standard.
• Adversarial Losses: HiFi-GAN, BigVGAN, LS-GAN, and feature-matching losses are used to improve perceptual quality, especially in GAN-based and hybrid architectures (FINALLY, UNIVERSE++) (Babaev et al., 2024, Scheibler et al., 2024).
• Score-Matching/Flow Objectives: Score-based diffusion uses denoising score-matching; flow models rely on matching the target field in continuous space.
• Content and Fidelity Losses: CTC-based phoneme loss, SpeechBERTScore, and LPS are included to preserve linguistic content and intelligibility.
• Plug-in Gating and Adaptation: Systems dynamically gate enhancement based on downstream task metadata, optimizing end-to-end metrics like ASR WER, SV EER, or embedding fidelity (Plugin-SE) (Chen et al., 2024).

3.2 Evaluation Metrics

• Intrusive: PESQ, POLQA, ESTOI, SI-SDR, SDR, LSD, MCD.
• Non-Intrusive: DNSMOS, NISQA, UTMOS, SCOREQ.
• Downstream: SpeechBERTScore, LPS (phone similarity), SpkSim (speaker cosine similarity), CAcc (character/ASR accuracy), WER.
• Subjective: MOS by P.808, CCR, and language-agnostic evaluations (Saijo et al., 29 May 2025, Li et al., 20 Jan 2026, Liu et al., 27 Jan 2026).

3.3 Datasets and Benchmarks

• Benchmarks are constructed by combining and filtering large speech corpora, noise sets, and RIRs (LibriTTS, VCTK, CommonVoice, DNS, AudioSet, FSD50K, WHAM!).
• Both simulated and real-recorded noisy utterances are used, spanning multiple languages (EN, DE, FR, ES, ZH, plus Japanese as an unseen language).
• Blind test sets stress domain and distortion generalization, including codec variations, packet loss, and multi-linguality (Li et al., 20 Jan 2026, Saijo et al., 29 May 2025).

The URGENT Challenge series (Interspeech/ICASSP 2025–2026) defines the universal SE task with standardized evaluation covering seven or more distortions, seven sampling rates, and multiple languages, with both objective and subjective benchmarks (Saijo et al., 29 May 2025, Li et al., 20 Jan 2026).

4. Key Results, Comparative Analyses, and Empirical Insights

Hybrid models that integrate discriminative signal mapping (masking, regression, or compression) with generative or code-token branches consistently outperform purely discriminative or generative models across metric categories.

4.1 Performance Highlights

System Type	PESQ ↑	ESTOI ↑	MOS (P.808) ↑	Downstream (e.g., WAcc) ↑	Distortion Generalization
Pure Discriminative	2.64	0.82	3.24	79.8%	Best on classical metrics, robust across languages but less 'natural' (Saijo et al., 29 May 2025).
Hybrid (Disc + Gen)	2.47–2.58	0.79–0.83	3.26–3.44	76–86%	Stronger MOS, perceptual quality, reduced artifacts (Liu et al., 27 Jan 2026, Saijo et al., 29 May 2025).
Pure Generative	1.96–2.34	0.74	3.43	~73%	Smoother, fails to preserve content/fidelity in unseen languages or packet loss (Liu et al., 27 Jan 2026).

Key observations:

• Purely generative models rank highest in subjective naturalness but are prone to content hallucination and language-dependence.
• Discriminative backbones are more robust to unseen speakers and languages.
• Hybrid or composite systems—combining predictive and generative modeling (e.g., PGUSE, TS-URGENet, FUSE, hybrid TF-GridNet+AR)—achieve leading or state-of-the-art performance by complementing strengths: discriminative for fidelity, generative for perceptual restoration and inpainting (Zhang et al., 30 May 2025, Liu et al., 27 Jan 2026, Rong et al., 24 May 2025, Goswami et al., 1 Jun 2025).

4.2 Subjective and Downstream Outcomes

In the URGENT 2025/2026 blind tests, generative and hybrid systems received higher MOS ratings (up to 3.44 vs. 3.24 for top discriminative), but the highest character accuracy, SI-SDR, and ESTOI remained with discriminative or hybrid systems (Saijo et al., 29 May 2025, Li et al., 20 Jan 2026).

Notably, naive increases in training data volume can degrade performance unless filtered for speech quality and MOS-based selection, highlighting the need for careful data curation.

5. Challenges, Open Problems, and Analysis

Several challenges and open research directions have been identified:

• Language dependency of generative models: Generative approaches have exhibited marked degradation on unobserved languages in both MOS and ASR metrics, suggesting that content-preserving regularization and explicit content losses (phoneme, LPS, SpeechBERTScore) are essential for multilingual universality (Saijo et al., 29 May 2025, Li et al., 20 Jan 2026).
• Bandwidth and artifact restoration: Standard regression models underperform on inpainting and bandwidth extension. Score-based diffusion and flow-matching models recover missing high frequencies more naturally but with higher computational cost.
• Inference speed and complexity: Diffusion and flow models require multiple (often 10–20) Euler steps. Truncation strategies and fusion with predictive initialization have reduced inference time to practical levels (e.g., as few as three reverse steps in PGUSE) (Zhang et al., 30 May 2025).
• Content preservation vs. perceptual quality: There is an inherent trade-off between maximizing MOS and maintaining ASR accuracy or phonetic content, especially under severe or compound distortions. Systems like UNIVERSE++ deploy phoneme-fidelity losses and LoRA fine-tuning to mitigate hallucination (Scheibler et al., 2024).
• Task-adaptive enhancement: Dynamic adaptation (as in Plugin-SE) aligns the enhancement process with downstream expectations, suggesting a path toward more flexible, plug-and-play universal modules (Chen et al., 2024).
• Self-supervised and multimodal universality: Pretraining universal encoders with masked autoencoding or audio-visual resynthesis (e.g., MAE-USE (Rajagopalan et al., 2 Feb 2026); ReVISE (Hsu et al., 2022)) enables zero-shot transfer, reduces dependence on pristine parallel data, and allows simultaneous enhancement and inpainting.

6. Future Directions and Research Opportunities

Recommendations for advancing universal speech enhancement systems include:

• Hybrid optimization and modular integration: Further research into adaptive, region-wise fusion between discriminative and generative branches may enhance both objective and perceptual metrics, improving handling of mixed or locally extreme degradations (Liu et al., 27 Jan 2026, Zhang et al., 30 May 2025).
• Semantics and content priors: Incorporating semantic guidance with content-aware tokens, text, or self-supervised units (e.g., SenSE, ReVISE) can preserve intelligibility and speaker identity, especially under information loss (Li et al., 29 Sep 2025, Hsu et al., 2022).
• Unsupervised, multi-domain, and multimodal learning: Pretraining on massive, untranscribed, and multi-condition corpora, and leveraging side information (video, text) can expand generalization and universality (Rajagopalan et al., 2 Feb 2026, Hsu et al., 2022).
• Real-time efficiency and deployment: Model compression, quantization, and inference acceleration, along with streaming-friendly architectures, are crucial for practical use.
• Unified evaluation protocols: Standardizing benchmarks, metrics, and subjective protocols—explicitly testing for both perceptual and content fidelity—will yield more informative and reliable comparisons (Li et al., 20 Jan 2026, Saijo et al., 29 May 2025).

Universal speech enhancement remains a rapidly evolving field driven by advances in deep learning, generative modeling, and self-supervised learning, with large-scale, community-driven challenges providing rigorous empirical baselines for future breakthroughs.