PS-TTS: Phonetic Synchronization in Text-to-Speech for Achieving Natural Automated Dubbing

Abstract: Recently, artificial intelligence-based dubbing technology has advanced, enabling automated dubbing (AD) to convert the source speech of a video into target speech in different languages. However, natural AD still faces synchronization challenges such as duration and lip-synchronization (lip-sync), which are crucial for preserving the viewer experience. Therefore, this paper proposes a synchronization method for AD processes that paraphrases translated text, comprising two steps: isochrony for timing constraints and phonetic synchronization (PS) to preserve lip-sync. First, we achieve isochrony by paraphrasing the translated text with a LLM, ensuring the target speech duration matches that of the source speech. Second, we introduce PS, which employs dynamic time warping (DTW) with local costs of vowel distances measured from training data so that the target text composes vowels with pronunciations similar to source vowels. Third, we extend this approach to PSComet, which jointly considers semantic and phonetic similarity to preserve meaning better. The proposed methods are incorporated into text-to-speech systems, PS-TTS and PS-Comet TTS. The performance evaluation using Korean and English lip-reading datasets and a voice-actor dubbing dataset demonstrates that both systems outperform TTS without PS on several objective metrics and outperform voice actors in Korean-to-English and English-to-Korean dubbing. We extend the experiments to French, testing all pairs among these languages to evaluate cross-linguistic applicability. Across all language pairs, PS-Comet performed best, balancing lip-sync accuracy with semantic preservation, confirming that PS-Comet achieves more accurate lip-sync with semantic preservation than PS alone.

• The paper introduces a phonetic synchronization-based TTS pipeline that achieves natural cross-lingual automated dubbing without video manipulation.
• It employs DTW-based vowel distance and joint COMET semantic scoring to ensure precise lip-sync and improved speech naturalness.
• Experimental results reveal significant improvements in LSE-D, LSE-C, and UTMOS metrics compared to baseline TTS and professional voice actors.

Phonetic Synchronization in Text-to-Speech for Cross-Lingual Automated Dubbing

Introduction

The proliferation of automated dubbing (AD) systems leveraging neural machine translation (NMT) and text-to-speech (TTS) models has shifted the focus from manual, voice-actor-driven workflows to automated pipelines. The primary challenge in AD is achieving high-fidelity isochrony and lip synchronization (lip-sync) across linguistically divergent languages without resorting to video modification techniques such as deepfakes. “PS-TTS: Phonetic Synchronization in Text-to-Speech for Achieving Natural Automated Dubbing” (2604.09111) presents a novel synchronization and paraphrasing pipeline, integrating phonetic and semantic alignment modules for robust AD, targeting Korean, English, and French language pairs.

Methodology Overview

Isochrony Module

The isochrony (ISO) process operates at the text level to ensure that paraphrased translations, generated via LLMs and NMT, align temporally with the source speech. The ISO process leverages:

• Phonemization and Tokenization: Source and target texts are converted to phoneme sequences with auxiliary blank tokens to facilitate alignment and improve naturalness.
• Pause Detection: Frame-level energy thresholding (RMS-based) and CTC-based forced alignment detect and refine utterance boundary intervals, robustifying synchronization in noisy environments.
• Speaking Rate Estimation and Target Text Selection: Token-wise speaking rates, estimated via a TTS duration predictor, underpin a constrained candidate paraphrase selection mechanism. Candidates satisfying both duration and cross-lingual semantic cosine similarity (via LaBSE embeddings) thresholds are retained; otherwise, iterative LLM-based paraphrasing is invoked.

Phonetic Synchronization (PS) Module

Critically, the PS module introduces a DTW-based vowel distance measure for candidate sentence selection:

• Vowel Distance Computation: Mean embeddings of vowel phonemes are projected and clustered (K-means, $N=5$) over source and target languages, enabling fine-grained cross-lingual comparison utilizing Euclidean centroid distances.
• DTW-Based Matching: For 60 paraphrased candidate sentences, the alignment cost is minimized with respect to the vowel sequence, prioritizing candidates that best preserve source-relevant mouth articulation.

PS-Comet: Joint Phonetic and Semantic Optimization

Acknowledging the potential orthogonality between phonetic and semantic constraints, PS-Comet fuses DTW-based phonetic alignment with COMET-driven semantic adequacy scoring using a convex combination. Empirical correlation analysis reveals weak dependency between these metrics, justifying their joint optimization.

Experimental Results

Datasets and Metrics

• Training: Libri-TTS-360 (English), KMSSS (Korean)
• Evaluation: Korean Lip-Reading Speech Recognition corpus, TCD-TIMIT, curated film-based voice-actor datasets
• Metrics: LSE-D (lip-sync error distance), LSE-C (lip-sync confidence), UTMOS (speech naturalness), VMAF (perceptual video quality)

Lip-Sync and Naturalness

Ablation studies demonstrate:

• Baseline TTS yields inferior synchronization compared to voice actors.
• ISO alone improves lip-speech alignment, but does not close the performance gap.
• PS-Comet achieves lower LSE-D and higher LSE-C than both baseline TTS and professional voice actors for Korean-English and English-Korean dubbing, with an LSE-D of 10.561 and LSE-C of 1.457, confirming its efficacy.
• On lip-reading datasets, ISO+PS-Comet achieves LSE-D=12.175 versus 12.671 (baseline) while UTMOS improves to 2.614 from 2.453.

Comparison to Deepfake and Multimodal Approaches

Application of PS-Comet-TTS, without altering source video frames, produces competitive lip-sync performance relative to deepfake-based synchronization (which scores best on LSE-D/C but degrades video quality and incurs high computational cost: 19min 29s per 10s video vs. 1min 34s for PS-Comet-TTS). PS-Comet also achieves higher VMAF, avoiding perceptual degradation.

Cross-Linguistic and Embedding Model Analysis

The approach generalizes to all pairs among Korean, English, and French, validating cross-linguistic applicability. In sentence selection, LaBSE embeddings consistently yield higher semantic similarity scores than SBERT or LASER.

Theoretical and Practical Implications

The introduction of explicit phonetic synchronization enables natural AD between structurally divergent languages—a scenario in which existing temporal or prosodic control methods (and deepfake-driven approaches) are less effective. The combination of DTW and COMET optimizes both visual and semantic congruence, decoupling the reliance on video manipulation and enabling efficient, scalable dubbing. The PS-TTS and PS-Comet-TTS architectures sidestep the video forgery risks and computational costs of deepfakes, while offering an interpretable mechanism for sentence selection.

The proposed pipeline’s modularity allows integration with more advanced LLMs, TTS engines, or semantic evaluators, underlining its extensibility. Methodological limitations include potential inadequacy of DTW and COMET to fully capture high-level fluency or poetic features, suggesting future augmentation with more sophisticated syntactic or prosodic metrics.

Conclusion

Phonetic Synchronization-based TTS (PS-TTS) and its joint semantic-phonetic variant PS-Comet-TTS constitute a robust, efficient pipeline for cross-lingual, lip-synchronized AD without recourse to video manipulation. Empirical evidence shows parity or superiority to professional dubbing in lip-sync and naturalness metrics across typologically dissimilar language pairs. The methodology is highly extensible and motivates further work incorporating advanced sentence selection metrics or direct multi-modal AV modeling for enhanced AD performance.