Synthetic Interleaved Speech-Text Data

• Synthetic interleaved speech-text data is a large-scale corpus with algorithmically interleaved speech and text segments, precisely aligned via advanced TTS models.
• It underpins multimodal model training by enabling SpeechLLMs and streaming TTS systems with enhanced diversity, optimized interleaving ratios, and efficient scaling.
• Generation pipelines combine randomized text sampling with controlled TTS conversion, achieving low latency, robust evaluation metrics, and improved cross-modal instruction following.

Synthetic interleaved speech-text data refers to large-scale corpora in which speech and text segments are algorithmically interleaved, with the speech components generated via text-to-speech (TTS) models and precise alignment maintained between spoken audio and textual content. These data support the training of multimodal models that process and generate both speech and text, and underpin current advances in speech LLMs (SpeechLLMs), streaming text-to-speech systems, and cross-modal instruction following.

1. Principles and Motivations

The scarcity and costliness of naturally occurring, well-aligned speech–text corpora have inspired the development of synthetic generation pipelines leveraging advances in neural TTS and LLMs. Synthetic interleaved speech–text data serves multiple roles:

• Scaling up pre-training of SpeechLLMs for instruction following, speech-language understanding, and end-to-end spoken dialog systems (Wang et al., 4 Mar 2025, Zeng et al., 2024, Maimon et al., 3 Apr 2025).
• Enabling streaming and real-time text-to-speech (TTS) by integrating acoustic and linguistic sequences into a unified autoregressive modeling framework (Wang et al., 14 Jun 2025, Yang et al., 2024).
• Providing near-unlimited, diverse training material for TTS research and on-device keyword spotting (Dua et al., 15 Sep 2025, Gan et al., 11 Nov 2025).

These pipelines systematically address data bottlenecks, allow for control over diversity (linguistic, phonetic, and speaker), and permit fine-tuned manipulation of text-to-speech/audio–text alignments in a way not possible with natural, manually curated corpora.

2. Data Generation and Interleaving Methodologies

Synthetic interleaved data construction typically starts with a large cleaned text corpus, from which segments are sampled and selectively synthesized into speech. For instance, InSerter (Wang et al., 4 Mar 2025) employs the following process:

1. Randomized text-segment sampling: From a filtered 610 billion-token corpus, contiguous word- or sentence-level spans are randomly chosen, mixing granularities to optimize downstream instruction-following—the best configuration uses approximately 30% word-level and 40% sentence-level tokens.
2. Text-to-speech conversion: Each segment undergoes TTS synthesis (CosyVoice 2.0), with acoustic diversity enforced via selection from 10,000 speaker "voice prompts" selected to minimize WER and maximize perceptual MOS.
3. Alternating construction: The training example is a sequence $[S_1, Y_1, S_2, Y_2, ..., S_N, Y_N]$, where $S_i$ are continuous speech embeddings (from a frozen speech encoder plus adapter) and $Y_i$ are text token sequences.
4. Loss and masking: Pre-training proceeds via cross-entropy on the text-only targets; speech embeddings are masked in the loss computation.

A distinct approach is found in scaling studies of interleaved LLMs (SIMS) (Maimon et al., 3 Apr 2025), where interleaved scheduling follows a Poisson process over word segments to achieve a targeted speech-to-text ratio, typically $η = 0.3$ (30% speech words). Segments are shuffled into [TEXT] and [SPEECH] blocks, with speech rendered as discrete HuBERT units.

Streaming TTS models such as StreamMel (Wang et al., 14 Jun 2025) and IST-LM (Yang et al., 2024) implement fixed-ratio interleaving at the block level (e.g., 1 phoneme token : 4 mel-spectrogram frames), supporting low-latency autoregressive synthesis and direct causal modeling of multimodal sequences.

Exemplary Interleaving Configurations

System	Segment Type	Ratio/Rationale	Speech Units	Evaluation Metric
InSerter	Word / sentence segment	30% / 40% (tokens)	TTS waveform	Instruction accuracy
SIMS	Poisson (λ=10) over words	30% speech, 70% text	HuBERT units	Semantic completion (tSC)
IST-LM	m BPE tokens ↔ n semantic codes	Optimal $m:n=1:3$	S3Tokenizer code	WER, speaker similarity
StreamMel	n phonemes ↔ m mel frames	$n:m=1:4$	mel-spectrogram	WER, Real-time factor

3. Scaling, Diversity, and Alignment

Scaling analysis demonstrates that interleaved synthetic corpora enable more compute- and parameter-efficient SpeechLLM training compared to textless or pure speech-only setups (Maimon et al., 3 Apr 2025, Zeng et al., 2024). Notably, the optimal compute allocation for such models shifts toward larger model size $N$ and reduced token budget $D$, with scaling exponents ($\alpha, \beta$) favoring parameter increases.

Diversity and phonetic coverage are explicitly quantified in pipelines such as SpeechWeave (Dua et al., 15 Sep 2025), leveraging Type-Token Ratio (TTR), moving average TTR, mean pairwise similarity metrics, and diphone coverage. SpeechWeave's pipeline increases grouped sentence diversity by 44–48% and diphone coverage by up to 17% relative to direct LLM prompting, supporting robust downstream TTS.

Precise speech–text alignment is systematically enforced either in the TTS synthesis itself (FSQ/mel-band timing in CosyVoice 2), with auxiliary forced-aligners (Montreal Forced Aligner), or via Dynamic Time Warping techniques when post-processing is necessary for correlated audio-visual data (TIMIT-TTS (Salvi et al., 2022)).

4. Architectures and Losses for Interleaved Sequences

The principal training paradigm is autoregressive modeling—left-to-right prediction over the concatenated sequence of text and speech representations, with loss assigned only on the relevant positions. In StreamMel, the input sequence

$$\mathbf{z} = [\,x_0,\dots,x_{n-1},\;y_0,\dots,y_{m-1},\;x_{n},\dots,x_{2n-1},\;y_{m},\dots,y_{2m-1},\;\dots\,]$$

is modeled by a causal Transformer decoder, with multi-task loss: regression (L₁+L₂) for mel frames, KL divergence for latent distributions, spectrogram flux for dynamism, and stop-prediction for streaming termination (Wang et al., 14 Jun 2025).

IST-LM (Yang et al., 2024) formalizes the training sequence as alternating text chunk $x_{0:m-1}$ and speech codes $y_{0:n-1}$, optimizing standard cross-entropy for token prediction and analyzing optimality with respect to text–speech distance, accessible future context, and precedence frequency. The $1:3$ text:speech ratio is established as empirically optimal for streaming TTS, yielding only a 6–8% relative WER deficit versus non-streaming baselines.

Recent works (Zeng et al., 2024, Noroozi et al., 2024, Wang et al., 31 Mar 2025) also highlight the use of pre-trained tokenizers (vector-quantized and semantic), masking strategies for data corruption, and the critical importance of alternating segment types for achieving robust cross-modal reasoning.

5. Practical Applications and Benchmark Results

Synthetic interleaved speech–text data underlies a broad range of applications:

• SpeechLLM instruction following: SpeechInstructBench, leveraging ~300k hours interleaved data, realizes prompt and instruction accuracies of 36.56% and 47.38% (word-level). Continuation-writing post-training yields further gains (Wang et al., 4 Mar 2025).
• Streaming TTS: StreamMel achieves first-packet latency of approximately 10 ms and real-time factors of 0.18, with comparable MOS and WER to offline systems (Wang et al., 14 Jun 2025). IST-LM demonstrates a relative WER gap below 8% to non-streaming SOTA at $1:3$ text-chunk:speech-chunk ratio (Yang et al., 2024).
• Multilingual TTS training: SpeechWeave enables a 10–48% diversity increase, 97%+ normalization, and significant WER reductions after fine-tuning: e.g. English WER 9.36% vs. baseline 15.37% (Dua et al., 15 Sep 2025).
• On-device keyword spotting: SynTTS-Commands achieves 99.5% English, 97.9% Chinese KWS accuracy with extreme resource trade-offs, verified on low-power hardware (Gan et al., 11 Nov 2025).
• Dialogue and conversational platforms: SpeechDialogueFactory demonstrates cost-effective production of dialogue corpora, with MOS comparable to human data and intelligibility WER as low as 1.72–2.36% (Wang et al., 31 Mar 2025).
• Deepfake detection and forensic benchmarks: TIMIT-TTS employs DTW alignment and multi-system synthesis, supporting robust multimodal fusion and cross-system forensic detection experiments (Salvi et al., 2022).

6. Future Directions and Open Issues

Ongoing research seeks to expand interleaved data frameworks to new modalities (e.g., video/audio/text), multi-lingual environments, and low-resource settings. Key areas include:

• Further optimizing interleaving ratios and chunk sizes for task- and architecture-specific trade-offs (Yang et al., 2024).
• Improving the match between synthetic and real speech (reducing vocoder artifacts, enhanced data augmentation) (Yang et al., 2024, Wang et al., 4 Mar 2025).
• Automated diversity and quality evaluation metrics, incorporating LLM-based judges for content, naturalness, and scenario alignment (Wang et al., 31 Mar 2025, Dua et al., 15 Sep 2025).
• Efficient use of compute, model size, and token budget in scaling large speech-text models, informed by empirical scaling laws (Maimon et al., 3 Apr 2025, Zeng et al., 2024).

Synthetic interleaved speech–text data forms the cornerstone of contemporary multimodal AI for speech and language, with advances in scalable generation, alignment, diversity, and interleaving methodology driving sustained improvements in both fundamental research and practical applications.