EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs

Abstract: Recent multimodal LLMs (MLLMs) have shown strong capabilities in perception, reasoning, and generation, and are increasingly used in applications such as social robots and human-computer interaction, where understanding human emotions is essential. However, existing benchmarks mainly formulate emotion understanding as a static recognition problem, leaving it largely unclear whether current MLLMs can understand emotion as a dynamic process that evolves, shifts between states, and unfolds across diverse social contexts. To bridge this gap, we present EmoTrans, a benchmark for evaluating emotion dynamics understanding in multimodal videos. EmoTrans contains 1,000 carefully collected and manually annotated video clips, covering 12 real-world scenarios, and further provides over 3,000 task-specific question-answer (QA) pairs for fine-grained evaluation. The benchmark introduces four tasks, namely Emotion Change Detection (ECD), Emotion State Identification (ESI), Emotion Transition Reasoning (ETR), and Next Emotion Prediction (NEP), forming a progressive evaluation framework from coarse-grained detection to deeper reasoning and prediction. We conduct a comprehensive evaluation of 18 state-of-the-art MLLMs on EmoTrans and obtain two main findings. First, although current MLLMs show relatively stronger performance on coarse-grained emotion change detection, they still struggle with fine-grained emotion dynamics modeling. Second, socially complex settings, especially multi-person scenarios, remain substantially challenging, while reasoning-oriented variants do not consistently yield clear improvements. To facilitate future research, we publicly release the benchmark, evaluation protocol, and code at https://github.com/Emo-gml/EmoTrans.

• The paper proposes EmoTrans, a benchmark with four tasks (ECD, ESI, ETR, NEP) to evaluate dynamic emotion transitions in multimodal settings.
• The paper utilizes 1,000 curated video clips with rich annotations from 12 scenarios, ensuring robust evaluation of emotion change detection and reasoning.
• The paper's experiments reveal that while current MLLMs can detect emotion changes, predicting precise transitions and causal reasoning remain significant challenges.

Authoritative Summary of "EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs" (2604.23348)

Motivation and Benchmark Overview

EmoTrans addresses a critical and underexplored aspect in multimodal affective computing: the modeling and evaluation of emotion transitions in dynamic, socially interactive contexts. While prior benchmarks have focused on static emotion classification, this work establishes a comprehensive benchmark to evaluate the dynamic evolution of emotions, including transitions, reasoning, and prediction, within multimodal videos. EmoTrans comprises 1,000 manually curated and annotated video clips from 12 real-world scenarios, with over 3,000 task-specific QA pairs, spanning single-person and multi-person settings. The benchmark introduces four progressive tasks: Emotion Change Detection (ECD), Emotion State Identification (ESI), Emotion Transition Reasoning (ETR), and Next Emotion Prediction (NEP), providing a structured framework for assessing various facets of emotion dynamics understanding in Multimodal LLMs (MLLMs).

Dataset Construction and Annotation Design

EmoTrans utilizes two publicly available multimodal video corpora, OpenHumanVid and CH-SIMSv2, as data sources, ensuring diversity in social situations (conversation, conflict, emergency, etc.). Annotation follows rigorous guidelines:

• Emotion Change Filtering: Only video samples with categorical emotion transitions (between distinct classes) based on observable multimodal cues (facial, vocal, contextual) are retained.
• Person Identification: Individuals are uniquely identified using visual and spatial attributes to ensure annotation consistency in multi-person scenes.
• Emotion Labeling: Seven discrete categories (Happiness, Sadness, Anger, Fear, Disgust, Surprise, Neutral) are used for labeling, capturing both initial and final states for transitions.
• Transition Reasoning: Natural language explanations grounded in multimodal evidence are required, excluding speculative inference.
• Emotion Change Localization: Precise timestamp annotation ensures temporal grounding of transitions.

The annotation pipeline involved affective computing experts and trained annotators, with high inter-annotator reliability (Fleiss's Kappa: 0.893 for initial, 0.857 for final states), supporting the validity of the benchmark. The dataset captures a rich spectrum of emotion transitions, with a predominance of Neutral and Happiness, and includes a substantial fraction of multi-person interactions (29.4%).

Benchmark Task Formulation

The benchmark decomposes emotion dynamics evaluation into four tasks, operationalized via structured QA-pair prompts:

1. Emotion Change Detection (ECD): Binary detection of emotional transition for each individual.
2. Emotion State Identification (ESI): Determination of emotions before and after transition (or static state).
3. Emotion Transition Reasoning (ETR): Generation of evidence-grounded natural language rationales explaining emotion shifts or stability.
4. Next Emotion Prediction (NEP): Anticipation of a person’s future emotion based on pre-transition video segment.

Each task increases in complexity, requiring deeper temporal modeling, cross-modal cue integration, and context-sensitive reasoning, especially when multiple individuals interact.

Experimental Evaluation and Quantitative Results

Comprehensive zero-shot evaluation of 18 state-of-the-art MLLMs (including both open-source and closed-source) was conducted. Metrics include classification accuracy and generative evaluation (BLEU-4, ROUGE-L, BERTScore, LLM-based semantic scoring).

• Coarse-Grained (ECD) Performance: Closed-source models (e.g., GPT-5.4, Gemini-3-Pro) consistently outperform open-source counterparts, achieving up to 80.6% (GPT-5.4) on single-person ECD.
• Fine-Grained (ESI, NEP) Performance: Performance drops dramatically: no model exceeds 50% average accuracy across ECD, ESI, NEP, and NEP accuracy peaks at just 36%. Multi-person settings exacerbate difficulty, with ESI-Exact accuracy falling below 22% for most models.
• Task Difficulty Gradient: The accuracy degrades progressively from ECD to NEP, reflecting increased complexity in capturing content, cause, and prediction of emotion transitions.
• Modality Effects: Incorporating audio modality does not guarantee performance boost; fully multimodal models often lag behind visual-text models, implicating alignment and cue selection bottlenecks.
• Reasoning Variants: "Thinking" model variants (enhanced reasoning pipelines) yield only marginal and inconsistent improvements, indicating perception and temporal grounding as dominant hurdles, not generic reasoning depth.
• Multi-Person Challenge: Performance degradation in multi-person scenarios is stark, e.g., ESI-Exact accuracy drops from 46.4% to 20.7% on Qwen3.5-122B.

These results concretely establish that current MLLMs, even at scale, can recognize when emotions change, but are substantially limited in modeling precise state transitions, causal reasoning, and anticipation in dynamic multimodal contexts.

Implications and Future Perspectives

The findings highlight a critical gap between the capabilities required for robust affective intelligence in social robots, dialogue agents, and HCI systems and what current MLLMs deliver. Precise modeling of temporal emotion dynamics, person-specific grounding, and integration of heterogeneous cues remain open challenges. From a practical standpoint, these deficiencies directly impair applications demanding sophisticated emotion tracking (e.g., adaptive counseling, mediated negotiation, empathetic agents). Theoretical implications underscore the necessity for architectures able to align, fuse, and reason over temporally-entangled multimodal signals—mere scaling or deepening reasoning is insufficient.

Avenues for future research include:

• Temporal Cross-Modal Alignment: Developing architectures for fine-grained, temporally-consistent alignment of cues across modalities.
• Interaction Modeling: Explicit modeling of interpersonal emotional influence and disentanglement in group scenarios.
• Benchmark Expansion: Inclusion of broader social contexts, extended durations, and more diverse affective phenomena.
• Protocol Refinement: Enhanced evaluation protocols for dynamic emotion understanding and more granular error analysis.

Conclusion

EmoTrans establishes a rigorous benchmark for multimodal emotion dynamics and provides quantitative evidence of substantial performance gaps in current MLLMs. The benchmark’s task-centric framework, high-quality data, and progressive evaluation uncover domain-specific bottlenecks, guiding future technical directions in affective computing and multimodal reasoning. The practical and theoretical challenges identified here present opportunities for advancing foundational emotional intelligence in AI systems.