Towards a Phonology-Informed Evaluation of Multilingual TTS

Abstract: Neural TTS systems can sound natural across languages, but naturalness does not guarantee the preservation of sound contrasts that distinguish words from their grammatical forms. Standard metrics like MOS do not test for this. We propose a classifier-based framework that audits TTS output against language-specific phonological patterns using human speech as a benchmark. Testing Assamese advanced tongue root (ATR) vowel harmony with Meta's MMS TTS, we show that a classifier trained on human speech transfers to synthesized speech with minimal loss. The faithfulness audit reveals that [+ATR] mid vowels are realized as [-ATR] in 1/3 tokens despite an underlying [+ATR] specification, a bias absent in human speech. At the word level, predicted ATR labels classify harmony more accurately than transcription labels, indicating a gap between intended and produced phonology. The framework offers task-specific diagnostics and generalizes to other phonological contrasts with measurable acoustic cues.

• The paper introduces a classifier-based framework for evaluating phonological faithfulness in multilingual TTS using Assamese ATR vowel harmony as a diagnostic testbed.
• The methodology reveals that logistic regression achieves around 82% cross-domain accuracy while nonlinear models show larger generalization gaps with dominant underproduction errors in mid [+ATR] vowels.
• The findings underscore the need for phonology-informed evaluation in TTS development, paving the way for more precise diagnostics in both segmental and word-level analyses.

Phonology-Informed Evaluation of Multilingual TTS: Diagnostic Assessment Using Assamese ATR Vowel Harmony

Introduction

Assessing neural text-to-speech (TTS) systems has traditionally emphasized perceptual naturalness and intelligibility through metrics such as MOS and WER. However, these global metrics are insensitive to whether generated speech accurately realizes language-specific phonological contrasts essential for grammatical and lexical distinctions—particularly when evaluation targets low-resource languages or morphophonological alternations such as vowel harmony. This work introduces a classifier-based framework for evaluating whether state-of-the-art multilingual TTS systems preserve fundamental segmental contrasts, using Assamese Advanced Tongue Root (ATR) vowel harmony as a rigorous diagnostic testbed (2607.01965).

Methodology

The core methodology involves acoustic-phonological transfer: classifiers trained on manually-annotated human speech are evaluated on synthesized TTS outputs, quantifying the alignment between intended and realized ATR categories both at the segmental (vowel) and word levels.

Human and TTS Corpora

Speech from 14 native Assamese speakers was recorded and segmented, with manual annotation of ATR status, vowel height, and backness. Parallel TTS outputs were generated with Meta’s MMS TTS system. Acoustic features included speaker-normalized F1, F2, F3, B1, and duration from midpoints of vowels. This approach ensured robust, cross-speaker generalization in the human corpus and controlled normalization for the single-voice TTS output.

Classification Tasks

Two complementary tasks were defined:

• Task 1: Segment-level ATR classification (binary) using logistic regression (LR) and random forest (RF) classifiers under four cross-domain conditions (human-to-human, human-to-TTS, TTS-to-human, TTS-to-TTS) to quantify domain generalization.
• Task 2: Word-level harmony classification (three classes: AgrYesMixNo, AgrYesMixYes, AgrNoMixYes) aggregating both acoustic and predicted phonological sequence features to probe higher-level morphophonological consistency.

A detailed phonological faithfulness audit triangulated classifier predictions with gold standard ATR annotations to distinguish systematic overgeneration ($-\to+$) from underproduction ($+\to-$) errors.

Results

Cross-domain ATR Classification

A key finding is that LR achieves near-identical accuracy between human-only and human-to-TTS transfer (≈82%), revealing stable acoustic-phonology mapping across domains for linear models. RF training (nonlinear models), while yielding higher within-domain accuracy (90.5%), incurs a larger generalization gap for TTS (74.7%), indicative of its susceptibility to speaker/domain-specific decision boundaries.

At the vowel-level, classification for high and low vowels /i/ and /ɑ/ is near-perfect, but mid [+ATR] vowels /e/ and /o/ show significantly reduced accuracy within TTS, mirroring acoustic ambiguity.

Phonological Faithfulness Audit

The faithfulness audit surfaces a strongly directional error imbalance for TTS, absent in human speech: a 7:1 ratio of underproduction ($+\to-$) to overgeneration ($-\to+$) errors for ATR in mid vowels (TTS: underproduction 14.2%, overgeneration 2.1%; Human: both ≈9%).

Figure 1: Per-vowel mismatch rate; highest TTS mismatch arises for mid [+ATR] vowels /e/ and /o/, unlike in human speech where /u/, /ɯ/, /ɛ/ show more misclassification.

This result is further concretized by spectrographic comparisons:

Figure 2: Spectrograms (0–1500 Hz) of leteku: In TTS, both /e/ F1 tracks cluster near/above the human [+ATR/−ATR] boundary, consistent with underproduction of the [+ATR] feature.

Word-level Harmony Classification

At the word-level, classifier performance using gold ATR annotations on human data is strong (macro-F1: 0.83), but drops precipitously in TTS transfer (macro-F1: 0.49 for A+B_gold). In contrast, replacing gold sequence features with predicted ATR (A+B_pred) more closely tracks TTS acoustic realities and substantially mitigates the transfer gap (TTS macro-F1: 0.62), but at the expense of category coherence—indicative of systematic divergence between intended and produced categories. These findings demonstrate that the phonological mismatch at the segmental level propagates nontrivially to grammatical alternations at the word-level.

Implications and Future Directions

The study demonstrates that contemporary massively multilingual TTS fails to reliably encode segmental contrasts even for typologically central phenomena like ATR harmony, despite high perceptual naturalness. Faithfulness errors are asymmetric and persist despite deterministic synthesis, most acutely affecting mid [+ATR] vowels in morphophonological contexts where grammatical alternation is obligatory. The audit framework enables quantitative, linguistically interpretable diagnostics unavailable from prevailing TTS metrics.

Broader implications include:

• For TTS development: Integrating phonology-informed evaluation can pinpoint systematic deviations relevant for linguistic fidelity, inclusion, and downstream educational or clinical applications, particularly for under-resourced and morphophonologically rich languages.
• For theoretical phonology: Automated, classifier-based audits offer scalable experimental tools to probe the articulatory and acoustic realization of featural contrasts in both human and synthesized speech, facilitating cross-linguistic generalization.
• For future research: Extending this paradigm to additional contrasts (e.g., tone, vowel length, nasality), more languages, and improved acoustic modeling could inform both model architecture and data collection priorities.

Limitations of dataset size and imbalanced coverage in TTS outputs caution against overgeneralization of specific error rates, but the general framework is adaptable and scalable.

Conclusion

Phonology-informed evaluation exposes a systematic underproduction of ATR distinctions in MMS TTS for Assamese, with segmental and word-level consequences undetectable by standard perceptual metrics. The classifier-based audit framework is extensible to other contrasts and languages with measurable acoustic cues, providing developers and linguists with principled, fine-grained diagnostics for multilingual TTS evaluation (2607.01965).