ProsodyEval: Prosody Diversity Evaluation

• The paper introduces DS-WED, a token-based distance metric that achieves a Pearson correlation of 0.77 with human judgments on prosodic variations.
• ProsodyEval comprises 1,000 synthetic utterances from seven zero-shot TTS systems, annotated with Prosody Mean Opinion Scores to systematically benchmark prosody diversity.
• Empirical findings reveal that traditional acoustic metrics underperform compared to DS-WED, emphasizing the need for perceptually aligned evaluation protocols in TTS.

ProsodyEval is a prosody diversity evaluation framework and annotated benchmark for assessing the diversity, naturalness, and perceptual quality of prosodic variation in synthesized speech. Prosody diversity refers to variation in pitch, rhythm, stress, and intensity—dimensions crucial for expressiveness and listener engagement in text-to-speech (TTS) systems. Historically, the field has lacked unified, perceptually aligned metrics and datasets for systematic evaluation of prosodic diversity. ProsodyEval addresses this gap by anchoring prosody diversity assessment in large-scale human opinion scores and introducing DS-WED (Discretized Speech Weighted Edit Distance), a token-based distance metric exhibiting high correlation with human judgments. The framework enables benchmarking of state-of-the-art zero-shot TTS systems, rigorous study of modeling factors influencing prosodic variation, and diagnosis of limitations in both models and evaluation protocols (Yang et al., 24 Sep 2025).

1. ProsodyEval Dataset and Annotations

ProsodyEval comprises 1,000 synthetic utterances generated from seven mainstream open-source zero-shot TTS systems, systematically covering three core generative paradigms: autoregressive (AR), non-autoregressive flow-matching (NAR-FM), and masked generative modeling (MGM). For each input prompt sampled from LibriSpeech test-clean and Seed-TTS test-en, five renditions are synthesized per system using distinct random seeds, resulting in a maximally diverse set of prosodic outputs. Errors (including misalignments) are filtered to enforce experimental consistency. Each sample is trimmed for silence via VAD.

Prosodic diversity is annotated by 20 expert raters with a Prosody Mean Opinion Score (PMOS) per utterance pair. All 10 possible pairwise comparisons across the five renditions are presented, rated on a five-point Likert scale ranging from 1 (nearly identical prosody) to 5 (clearly and consistently different). Ratings total 2,000 high-quality scores, averaging two ratings per pair. Inter-rater agreement, as measured by Pearson correlation confidence intervals and consistency across groups, indicates high annotation reliability despite the absence of an explicit $\kappa$ coefficient (Yang et al., 24 Sep 2025).

2. Limitations of Prior Objective Prosody Diversity Metrics

Prevailing objective metrics for prosody diversity—such as log $F_{0}$ RMSE, mel-cepstral distortion (MCD), and trajectory-based variance or range of $F_0$ and durations—fail to holistically quantify prosodic variation:

• Partial prosodic coverage: Pitch-centric metrics ignore rhythm and intensity; MCD is limited to spectral envelope.
• Poor alignment with perception: These scores exhibit weak correlation with human judgments of prosody diversity.
• Computational complexity: Dynamic time warping (DTW) or similar approaches are required for temporal alignment of prosody trajectories with unequal lengths (Yang et al., 24 Sep 2025).

A plausible implication is that TTS system comparisons predicated on such metrics may be misleading in terms of perceived prosodic richness.

3. DS-WED: Discretized Speech Weighted Edit Distance

DS-WED is formulated to directly address these deficiencies by operating in a semantically rich token space derived from self-supervised learning (SSL) representations. The calculation involves:

• Tokenization: Silence-trimmed waveforms $\mathbf{X}_{1}$ and $\mathbf{X}_{2}$ are mapped via a SSL encoder (HuBERT or WavLM, typically at layer 8) to frame-wise embeddings, then clustered by $k$-means (commonly $k=50$) to obtain discrete semantic token sequences $\mathbf{c}_{1}$ and $\mathbf{c}_{2}$.
• Weighted Edit Distance: DS-WED computes the minimum-cost sequence of edit operations (substitutions, insertions, deletions) to align $\mathbf{c}_{1}$ and $\mathbf{c}_{2}$:

$$\mathrm{DS\textrm{-}WED}(\mathbf{c}_{1},\mathbf{c}_{2}) = \min_{\pi\in\mathcal{A}(\mathbf{c}_{1},\mathbf{c}_{2})} \sum_{(i,j,o)\in\pi} w_{o}\,c_{o}\!\bigl(c_{1,i},c_{2,j}\bigr)$$

with $w_{\mathrm{sub}}=1.2$ and $w_{\mathrm{ins}}=w_{\mathrm{del}}=1.0$, incorporating the perceptual insight that listeners are more sensitive to substitutions in prosody than to insertions/deletions.

The DS-WED measure is robust to choice of backbone (HuBERT, WavLM), cluster size, and SSL layer, peaking in performance around HuBERT layer 8 and $k=50$ clusters (Yang et al., 24 Sep 2025).

4. Correlation of DS-WED and Acoustic Metrics with Human Judgment

Empirical correlation analysis on the ProsodyEval dataset demonstrates that DS-WED substantially outperforms previous objective metrics in aligning with PMOS. Pearson correlation coefficients (aggregated over groups via Fisher’s $Z$ transform) are:

Metric	$\bar{r}$	95% CI
DS-WED	0.77	[0.73, 0.81]
MCD	0.66	[0.58, 0.73]
log $F_0$ RMSE	0.30	[0.19, 0.40]

DS-WED’s superiority is robust to SSL backbone, clustering parameters, and sequence length. Existing acoustic metrics, by contrast, show major perceptual misalignment and intrinsic insensitivity to prosodic aspects beyond spectral envelope or pitch (Yang et al., 24 Sep 2025).

5. Benchmarking TTS Systems and Modeling Factors

ProsodyEval and DS-WED have been used to provide a comparative benchmark across prominent TTS architectures:

System	DS-WED$_{\mathrm{LS}}$	Rank$_{\mathrm{LS}}$	DS-WED$_{\mathrm{Seed}}$	Rank$_{\mathrm{Seed}}$
XTTS-v2	127.8	4.89	93.2	5.50
CosyVoice	120.6	4.59	75.7	4.85
CosyVoice2	134.3	5.38	88.0	5.78
E2 TTS	84.9	2.11	52.4	2.18
F5-TTS	79.6	1.50	49.0	1.51
ZipVoice	114.5	3.93	58.6	2.88
MaskGCT	139.8	5.61	80.4	5.30

Key findings indicate that AR and MGM paradigms yield greater prosody diversity than flow-matching NAR (which produces monotonic, over-smoothed prosody due to implicit alignment). Duration control at inference (e.g., scaling durations by 0.8–1.2) significantly improves diversity in NAR systems, confirming the importance of duration as a diversity component. Reinforcement learning via Direct Preference Optimization, while beneficial for intelligibility, tends to reduce DS-WED scores, implying a trade-off between prosodic variation and clarity. Large audio LLMs (LALMs) such as Gemini 2.5 Pro currently demonstrate weak alignment to human assessments of prosodic difference ($\bar{r}=0.27$ with PMOS, $\bar{r}=0.22$ with DS-WED), underscoring the need for continuing perceptually grounded evaluation (Yang et al., 24 Sep 2025).

6. Related Human and Objective Evaluation Frameworks

Parallel efforts have explored prosody diversity and transfer in different modalities and feature spaces:

• Rich Prosody Diversity Modelling with Phone-level Mixture Density Network: Du and Yu propose a GMM-based phone-level mixture density network (GMM-MDN) but rely exclusively on log-likelihood of held-out embeddings and subjective AB-preference protocols to evaluate diversity. No explicit, numeric objective diversity metric is defined. Subjective results indicate clear user preference for GMM-MDN over single-Gaussian or utterance-level baselines (preference rates $\sim$70–90%), but listener statistics and significance tests are not specified (Du et al., 2021).
• ADEPT: The ADEPT dataset provides a prosody transfer evaluation protocol focused on classification of perceptually distinct prosodic renditions (emotion, attitude, focus, phrasing) in natural speech and TTS. Forced-choice classification by qualified listeners quantifies recognizability of prosodic categories, producing accuracy scores and confusion matrices. ADEPT emphasizes perceptual validity by selecting only classes and utterances with ≥60% recognition in pretesting and recommends $n=30$ qualified listeners per test, statistical binomial tests against chance, and public reporting of natural speech and TTS system results for transparency (Torresquintero et al., 2021).

A plausible implication is that ProsodyEval and DS-WED complement these methods by extending objective evaluation to token‐based, SSL-derived discrete spaces, while retaining close perceptual alignment.

7. Limitations, Open Problems, and Future Directions

ProsodyEval remains limited to English and has yet to validate DS-WED’s cross-lingual generalization. Current tokenization is optimized for English (fixed $k$-means clusters), which may not transfer to typologically distant languages or varied speaking styles. The operation weights in DS-WED are static and hand-set; data-driven learning of $w_{o}$ could enhance perceptual sensitivity. DS-WED is fundamentally a local, token-alignment measure and may miss higher-level or sequence-global prosodic structures such as pitch accent groups or phrase-level rhythm. Future metrics may integrate both edit-distance–style and continuous prosody-embedding approaches. Establishing language-agnostic, adaptive clustering and formalizing subjective protocols (e.g., listener pool statistics, significance thresholds) are further recommended to support transparent benchmarking and generalize evaluation (Yang et al., 24 Sep 2025).

The field thus continues to require standardized, reproducible, and perceptually aligned frameworks—such as ProsodyEval—for comparative analysis and model development in prosody-aware speech synthesis.