- The paper introduces a dual-layer evaluation framework that integrates situationally framed subjective assessments using Best Worst Scaling with objective ASR-based and distributional metrics.
- The paper demonstrates strong alignment between subjective judgments and objective metrics, with correlations (e.g., ρ up to 0.86) validating the dual-reference TTSDS2 score.
- The paper reveals a trade-off between improved intelligibility and preserved speaker identity, highlighting the need for adaptive fine-tuning in TTS for disordered speech.
Evaluation Frameworks for TTS Voice Reconstruction: Objective and Subjective Strategies
Problem Definition and Context
Voice reconstruction via TTS offers vital support for individuals with speech disorders, particularly in augmentative and alternative communication. The primary challenge is retaining speaker identity while maximizing intelligibility, a nontrivial task due to disorder-induced variability and absent "pre-condition" ground truth. Historically, assessment protocols have relied on general-purpose subjective metrics such as MOS, but these lack both sensitivity and task-relevance, motivating the need for more nuanced evaluation strategies.
Methodology
The paper introduces a dual-layered evaluation framework for TTS voice reconstruction, encompassing both subjective and objective approaches. The subjective layer utilizes Best Worst Scaling (BWS) with situational framing, instructing listeners to evaluate either intelligibility (INTELLIGIBILITY) or the balance between intelligibility and reconstructed speaker identity (RECONSTRUCTION). This design is directly informed by prior work on task-specific framing [18], addressing bias observed in unframed evaluations.
Objective evaluation incorporates several canonical metrics:
- ASR-based intelligibility: WER, PER, and UTMOS (automatically predicted MOS).
- Speaker similarity: Cosine similarity between speaker embeddings extracted from synthetic and reference speech.
- Distributional evaluation: The TTSDS2 metric, approximating trade-offs by comparing synthesized speech distributions to (a) a high-intelligibility corpus (LibriTTS) and (b) the original disordered recordings. The mean of these distances yields a composite score reflecting the dual task objectives.
Experiments span 17 zero-shot TTS models and 193 speakers from the SAP dataset [30], sampled across four disorder types and two intelligibility strata.
Empirical Results
Subjective BWS evaluations reveal that:
- Intelligibility: Most TTS systems (e.g., StyleTTS2, Fish Speech, Open Voice) surpass original recordings, especially for severely disordered speech. StyleTTS2 delivers the highest intelligibility.
- Reconstruction: Only a few systems (IndexTTS2, Qwen3-TTS, E2-TTS) outperform recordings when balancing intelligibility and identity, particularly for low-intelligibility speakers. Many models improve intelligibility at the cost of speaker characteristics, underscoring a critical trade-off.
Objective measures:
- ASR-based metrics (WER, PER, UTMOS) correlate strongly with subjective intelligibility (Spearman ρ up to 0.86 for UTMOS).
- Speaker embedding similarity aligns moderately with subjective reconstruction rankings, but is confounded, as embeddings encapsulate both speaker identity and disorder characteristics.
- Distributional TTSDS2 scores (especially their mean) demonstrate strong correlation with subjective reconstruction judgments, outperforming speaker similarity for both all-speaker and low-intelligibility subsets (ρ = 0.81 and 0.73 respectively).
A salient numerical result is that the top-performing systems according to the proposed dual-reference TTSDS2 score precisely match those favored in subjective reconstruction tasks, validating the metric's practical alignment.
Implications and Future Directions
The results reinforce the inadequacy of generic naturalness scores (MOS) for voice reconstruction, supporting a paradigm shift to task-specific, situationally framed evaluations. The proposed framework enables robust differentiation among TTS models in practical deployment scenarios, and provides baseline reference systems (IndexTTS2, Qwen3-TTS, E2-TTS) for benchmarking.
Distributional evaluation, especially dual-reference metrics, offers a scalable and objective proxy for subjective listener preferences. This has major implications for AI-powered speech technology research:
- Streamlining model selection for clinical VOCA deployment.
- Facilitating rapid, repeatable evaluation across new TTS architectures and disorder types, including non-English contexts.
The findings surface theoretical insights into the limits of zero-shot TTS for severely disordered speech: As disorder severity increases, systems trend towards generic high-intelligibility, undermining identity preservation. This accentuates the necessity for adaptive or fine-tuning strategies, perhaps integrating disorder-specific features or utilizing multi-modal datasets for improved disentanglement.
Extensions to user-centric evaluation, prosody, accent, or usability tasks are natural next steps. Future work should focus on broadening dataset diversity and incorporating direct VOCA user preference assessment.
Conclusion
This paper presents a comprehensive, scalable evaluation framework for TTS voice reconstruction, demonstrating the superiority of situationally framed subjective assessments combined with distributional objective metrics. The dual-reference TTSDS2 approach reliably quantifies the crucial intelligibility-identity trade-off, aligning closely with human judgments. These methodologies address key gaps in TTS evaluation and establish both practical and theoretical foundations for future AI research in clinical and assistive speech synthesis (2606.21343).