EmoTransCap: Dataset and Pipeline for Emotion Transition-Aware Speech Captioning in Discourses

Abstract: Emotion perception and adaptive expression are fundamental capabilities in human-agent interaction. While recent advances in speech emotion captioning (SEC) have improved fine-grained emotional modeling, existing systems remain limited to static, single-emotion characterization within isolated sentences, neglecting dynamic emotional transitions at the discourse level. To address this gap, we propose Emotion Transition-Aware Speech Captioning (EmoTransCap), a paradigm that integrates temporal emotion dynamics with discourse-level speech description. To construct a dataset rich in emotion transitions while enabling scalable expansion, we design an automated pipeline for dataset creation. This is the first large-scale dataset explicitly designed to capture discourse-level emotion transitions. To generate semantically rich descriptions, we incorporate acoustic attributes and temporal cues from discourse-level speech. Our Multi-Task Emotion Transition Recognition (MTETR) model performs joint emotion transition detection and diarization. Leveraging the semantic analysis capabilities of LLMs, we produce two annotation versions: descriptive and instruction-oriented. These data and annotations offer a valuable resource for advancing emotion perception and emotional expressiveness. The dataset enables speech captions that capture emotional transitions, facilitating temporal-dynamic and fine-grained emotion understanding. We also introduce a controllable, transition-aware emotional speech synthesis system at the discourse level, enhancing anthropomorphic emotional expressiveness and supporting emotionally intelligent conversational agents.

• The paper introduces a framework for emotion transition-aware speech captioning that models dynamic shifts in conversational emotion.
• It utilizes a two-stage automated pipeline combining acoustic analysis with LLM-based semantic synthesis to generate discourse-level captions.
• Empirical results demonstrate significant gains in emotion detection accuracy and TTS naturalness, outperforming previous state-of-the-art methods.

Emotion Transition-Aware Speech Captioning: The EmoTransCap Dataset and Pipeline

Motivation and Problem Formulation

Automatic emotional understanding and controllable expressive speech generation are central challenges for open-domain human-agent interaction, demanding dynamic affect perception and nuanced emotional synthesis. Prior speech emotion captioning (SEC) research has focused almost exclusively on single-utterance, single-emotion scenarios, with both models and datasets ignoring natural, temporally-evolving emotion transitions within discourse. This static assumption conflicts with psychological evidence that emotional states evolve fluidly across conversational turns, resulting in significant informational loss for systems designed for authentic empathetic dialog.

The EmoTransCap framework directly targets this deficiency. The core contributions are: 1) formulation of the emotion transition-aware speech captioning task operating on discourse segments with multiple, temporally-constrained emotional transitions, and 2) provision of the first large-scale, bilingual (English and Chinese) synthetic speech dataset, EmoTransSpeech, annotated with aligned, discourse-level natural language emotion transition captions. Furthermore, the pipeline enables the first realization of controllable, temporally-aware emotional TTS at the discourse level.

The foundational concept of modeling emotion transitions within discourse is illustrated in the figure below.

Figure 1: EmoTransCap concept—accurate encoding of emotion-transition cues at the discourse level (example with one transition).

Dataset Generation and Structure

Automated Pipeline

The EmoTransSpeech dataset is generated through a two-stage automated pipeline, balancing scalability and annotation fidelity. Figure 2 presents a schematic overview of the process.

Figure 2: Pipeline for discourse-level emotion transition-aware speech dataset construction. Gray waveforms indicate silent segments.

• Stage 1: Topic and Discourse Generation A hierarchical topic taxonomy ensures broad semantic coverage, with primary categories such as Business, Culture, Science, etc., and 420 manually-vetted subtopics. For each, dialogic discourse is synthesized using Gemma-3, controlling for emotion transitions (1–3 per discourse), basic emotion categories (angry, happy, sad, neutral, surprised), and narrative perspective. This yields 420 unique emotion transition type combinations.
• Stage 2: Speech Synthesis and Consistency Checking CosyVoice2, a zero-shot TTS system, is used for sentence-wise emotional speech generation, conditioned on reference emotional prompts drawn from ESD and RAVDESS. Generated speech undergoes automated consistency checking using emotion2vec representations, ensuring alignment to target emotion labels. Discourse-level continuity is enforced via loudness normalization post-concatenation.

The final dataset comprises 144,000 discourses (over 617 hours), evenly split across male/female English and Mandarin speakers, with natural variation in utterance and caption lengths as shown in the following distribution figures.

Figure 3: Age distribution of speakers within the EmoTransSpeech dataset.

Figure 4: Distribution of caption lengths for different SEC models.

Annotation Pipeline and Model Architecture

The fully-automated annotation pipeline—outlined in Figure 5—yields fine-grained, emotion transition-aware natural language captions by integrating acoustic analysis, temporal diarization, and LLM-based semantic synthesis.

Figure 5: End-to-end workflow of the EmoTransCap annotation pipeline with a 1-transition discourse example.

• Speech Pre-processing Silence regions are eliminated using WebRTC-based VAD, with ASR (whisper-large-v2) providing transcript alignment for downstream processing.
• Multi-Task Emotion Transition Recognition (MTETR) The core diarization module leverages emotion2vec embeddings fed through a ResNet-Transformer-BiLSTM stack, jointly trained for emotion transition detection and diarization using a multi-task uncertainty-weighted loss. Ablations confirm both auxiliary ETDet loss and sequence modeling modules are critical for effective boundary and context modeling.
• Acoustic Attribute Analysis Segment-level analysis generates explicit style/prosody descriptors (pitch, energy, speed, etc.) following the paradigm in SpeechCraft.
• Emotion Cues Integration and Caption Generation Outputs from previous stages condition Gemma-3 for dual-format captioning: (a) Descriptive ($V_D$)—rich global/segmental summaries for SEC and multimodal understanding, and (b) Instructional ($V_I$)—structured SSML-like directives suitable for direct controllable speech synthesis.

Experimental Results

Dataset and Model Quality

In subjective MOS evaluations across 30 random samples, the dataset achieves perfect or near-perfect accuracy in emotion transition count/type detection and high semantic and perceptual MOS scores (ranging 4.33–4.79 across one to three transitions). The automatically-generated captions are consistently aligned with the underlying emotional and acoustic dynamics, confirming the utility of the dual-caption strategy.

Supervised Model Performance

For discourse-level SEC, fine-tuned EmoTransCap models ($V_I$/$V_D$) considerably outperform prior SOTA baselines (SECap, SpeechCraft). Notably, EmoTransCap ($V_I$) reaches 100% accuracy in both emotion transition count and type across all transition conditions, compared to ≤36.84%/22.0% for fine-tuned SECap/SpeechCraft and 0% for raw models, with 3–4.7 MOS in terms of caption-speech consistency. These results clearly demonstrate the criticality of transition-aware resources for effective SEC.

Emotion Expression: Discourse-level TTS

Using EmoTransCap data for TTS adaptation, controllable CosyVoice2 achieves substantial gains in emotional consistency (MOS-E increases from <2.36 to 4.88) and naturalness (MOS-S from <1.83 to 4.58), and objective emotion embedding similarity (EES$_{ET}^3$ reaches 49.16%) relative to non-fine-tuned models. Format-1 prompt structures show better emotional alignment compared to global prompts (Format-2), suggesting the importance of segment-level prompt-cue alignment for high-fidelity TTS control.

Implications and Future Research

EmoTransCap sets a new task and resource standard for dynamic emotion modeling in spoken dialogs, directly enabling:

• More accurate, temporally-resolved emotional understanding for empathetic dialog agents.
• Controllable emotional TTS capable of conveying nuanced emotional transitions, essential for natural interaction in assistive, educational, and affective computing applications.
• Unified discourse-level benchmarks, permitting rigorous evaluation and development of neural architectures for emotion transition modeling.

Future directions include expansion to naturalistic (non-synthetic) dialog data, automated evaluation metrics harmonized for temporal emotion dynamics, and extension to multimodal affect cues (e.g., facial expression, physiological signals) for holistic emotion perception and synthesis.

Conclusion

EmoTransCap establishes a new framework and resource for discourse-level emotion transition-aware speech captioning and synthesis. Its comprehensive dataset, modular annotation pipeline, and strong empirical results demonstrate the necessity and feasibility of dynamic emotion modeling for both perception and expressive generation. Ongoing work is required to close the gap with natural human emotional expressiveness and to enrich the evaluation of spoken emotional intelligence.

For further reading and replication, see the complete text of "EmoTransCap: Dataset and Pipeline for Emotion Transition-Aware Speech Captioning in Discourses" (2604.26417).