Speech Emotion Recognition (SER)

• Speech Emotion Recognition (SER) is the task of inferring emotions from speech by analyzing acoustic cues, linguistic content, and self-supervised representations.
• Systems utilize deep architectures such as CNNs, LSTMs, and transformers to model temporal dynamics and speaker-specific traits for enhanced performance.
• Research in SER focuses on multimodal fusion, transfer learning, and robust evaluation to address challenges like speaker variability and noisy conditions.

Speech Emotion Recognition (SER) is the task of inferring affective state from speech, most commonly as utterance-level classification into discrete emotion categories, though dimensional formulations such as valence, arousal, and dominance also appear in contemporary systems. Across the literature, SER is framed as a problem of extracting emotion-relevant information from spectral envelope, prosody, timing, voice quality, lexical content, speaker-dependent expressive patterns, and, more recently, self-supervised and physiology-informed representations of speech production. The field spans hand-crafted acoustic descriptors, spectrogram-based deep models, speaker-aware attention mechanisms, self-supervised speech encoders, multimodal speech–text systems, and production-oriented physiological modeling (Wang et al., 2019, Li, 25 Jan 2026, Zhang et al., 3 Feb 2026).

1. Task formulation and emotional information in speech

SER is generally formulated as supervised multi-class classification from speech audio alone or from speech combined with auxiliary modalities. Representative categorical taxonomies include four-class settings such as happy, neutral, angry, and sad on IEMOCAP, seven-class settings such as angry, disgust, fear, happy, pleasant surprise, sad, and neutral on TESS, and larger multilingual taxonomies in corpora such as RAVDESS, Emo-DB, SHEMO, MSP-Podcast, and ShEMO (Wang et al., 2019, Oluwademilade et al., 16 Apr 2026, Jafarzadeh et al., 2024, Wu et al., 2024). Dimensional variants also appear in ASR-integrated work, where valence, arousal, and dominance are predicted alongside categorical labels (Li, 25 Jan 2026).

The acoustic basis of SER is consistently described in terms of emotion-dependent modulation of pitch, energy, timing, rhythm, speaking rate, spectral structure, formants, timbre, and voice quality. Several works emphasize that these cues unfold over time, making temporal modeling central rather than optional. Emotional ambiguity, speaker variability, recording conditions, lexical dependence, and background noise are recurring sources of difficulty, and they are repeatedly identified as the main obstacles to robust generalization outside controlled corpora (Oluwademilade et al., 16 Apr 2026, Jafarzadeh et al., 2024, Li, 25 Jan 2026).

A further complication is that emotional expression is not speaker-invariant. Work on speaker-attentive SER explicitly argues that individuals use different means to express emotions, and conditions the emotion recognizer on speaker identity through a Self Speaker Attention mechanism (Moine et al., 2021). Other studies broaden the signal basis beyond acoustics, arguing that lexical content, ASR hidden states, phonation excitation, articulatory kinematics, instantaneous frequency, and group delay all encode complementary affective information that conventional audio-only pipelines may underuse (Li, 25 Jan 2026, Zhang et al., 11 Nov 2025, Zhang et al., 3 Feb 2026).

2. Feature representations and preprocessing strategies

A large fraction of SER systems rely on hand-crafted acoustic representations. MFCCs remain a dominant baseline because they approximate human auditory perception, compactly summarize the spectral envelope, and pair naturally with sequence models. In one representative formulation, the Mel transform is written as $m = 2595 \log_{10}(1 + f/700)$, and the MFCC pipeline is described as STFT or power spectrum, Mel filterbank energies, log energies, and DCT (Oluwademilade et al., 16 Apr 2026). Other works extend the feature space with deltas and accelerations, pitch- and energy-related descriptors, zero-crossing rate, spectral centroid, spectral contrast, spectral roll-off, RMS, eGeMAPS, ComParE_2016, IS10_Paraling, and large openSMILE functional sets (Wang et al., 2019, Yazdani et al., 2022, Nigar, 2024, Kundu et al., 2024).

Time–frequency representations remain equally central. Mel-spectrograms and log-mel spectrograms are widely used as 2D inputs to CNNs and CNN–RNN hybrids, and several studies explicitly exploit multiple resolutions. The dual-sequence LSTM model preprocesses each utterance into MFCCs plus two mel-spectrograms computed with 256-point and 512-point FFTs, using the time–frequency trade-off to jointly represent transient prosodic events and more precise spectral structure (Wang et al., 2019). Other architectures use 26-filter MFCC maps coerced to 1.8 s windows, or 128-band log-mel spectrograms resized as image-like tensors for CNN transfer learning (Tang et al., 2024, Vu, 26 Aug 2025).

Variable-length handling differs markedly across studies. Some pipelines trim or pad clips to fixed durations such as 3 s or 7.52 s; others segment utterances into overlapping windows; others avoid truncation through statistics pooling. The ResNet34 transfer-learning system for IEMOCAP uses a statistics pooling layer that concatenates the temporal mean and standard deviation of frame-level activations, enabling training on random chunks and evaluation on the full utterance without sequence truncation (Padi et al., 2021). The dual-resolution DS-LSTM work instead uses nearest-neighbor interpolation to the median time length for mel-spectrograms, followed by logarithmic transformation, arguing that this avoids information loss due to truncation and avoids the computational overhead of padding (Wang et al., 2019).

A more recent shift is the use of raw waveform inputs with frozen self-supervised speech encoders. Wav2Vec 2.0 and HuBERT have been used as feature extractors from raw audio, producing frame-level embeddings that are mean-pooled into 768-dimensional utterance vectors and classified by a lightweight feed-forward head (Jafarzadeh et al., 2024). This line of work reduces dependence on hand-crafted front ends and suggests that a substantial part of SER can be recast as downstream learning over pretrained speech representations.

3. Recurrent, convolutional, and attention-based acoustic modeling

Early deep SER work commonly treated spectrograms as images and learned local patterns with CNNs. A notable example is the Deep Retinal Convolution Neural Networks approach, which paired a spectrogram-based DCNN with a scale-based augmentation scheme inspired by retinal imaging and reported very high within-study accuracies on IEMOCAP, EMO-DB, and SAVEE (Niu et al., 2017). Later studies moved toward architectures that explicitly separate local and temporal structure, such as CNN+LSTM, CNN+BiLSTM, CRNN, and CNN–Transformer hybrids (Wang et al., 2019, Tang et al., 2024, Kundu et al., 2024).

Sequence models remain especially prominent when MFCCs or frame-level low-level descriptors are used as inputs. On TESS, a unidirectional LSTM over 40-dimensional MFCC sequences achieved 99% test accuracy, compared with 98% for an RBF SVM on time-averaged MFCCs, under a speaker-dependent 80/20 split (Oluwademilade et al., 16 Apr 2026). On ShEMO, BLSTM and Attention-BLSTM models outperformed feed-forward DNNs on frame-level low-level descriptors, and CNN+Attention-BLSTM was strongest at 32 ms frame resolution, reaching 63.52% UA and 75.32% WA with the hand-crafted 52-feature set (Yazdani et al., 2022).

The DS-LSTM architecture introduced a more explicit multi-stream recurrent mechanism. It processes two synchronized spectrogram sequences with a recurrent cell that adds a second input gate and a second candidate cell, yielding six gates in total. One candidate cell is biased toward time-rich cues and the other toward frequency-rich cues, and recurrent batch normalization is applied to stabilize training. Combined with a separate MFCC-LSTM branch and late score averaging, the model reached 72.7% WA and 73.3% UA on four-class IEMOCAP, outperforming the paper’s unimodal baselines and approaching multimodal audio+text performance (Wang et al., 2019).

Attention mechanisms have progressively moved from generic temporal weighting to more structured designs. The CNN–Transformer architecture with multidimensional attention uses irregular 3×1 and 1×3 convolutions for local extraction, a time-channel-space attention module with BiLSTM-based temporal attention plus Shuffle Attention over spatial and channel dimensions, and a Lightweight Convolution Transformer block for global context. Under the authors’ random 80/20 IEMOCAP split, it achieved WA = 71.64 and UA = 72.72, with ablations showing that removing temporal attention or the local–global fusion block reduced both metrics (Tang et al., 2024). Similarly, the Efficient Channel Attention guided CNN-BiLSTM framework combines attention-based local feature blocks with two BiLSTM global feature blocks over fused MFCC, mel-spectrogram, RMS, and ZCR inputs, reporting mean accuracies of 99.65% on TESS, 94.88% on RAVDESS, 98.12% on BanglaSER, 97.94% on SUBESCO, and 97.19% on Emo-DB (Kundu et al., 2024).

Alternative representational ideas also appear. Quaternion CNNs encode RGB mel-spectrogram images in the quaternion domain and use quaternion convolutions to model inter-channel dependencies, reporting 77.87% accuracy on RAVDESS, 70.46% on IEMOCAP, and 88.78% on EMO-DB, while also reducing model size relative to real-valued Deep-CNN and CNN-LSTM baselines (Muppidi et al., 2021).

4. Speaker awareness, transfer learning, and self-supervised speech encoders

Speaker conditioning is one of the clearest attempts to address inter-speaker heterogeneity directly. The speaker-attentive SSA-CRNN framework couples a speaker recognition ACRNN with an emotion recognition ACRNN, injecting BLSTM speaker features into the emotion branch through query-key-value attention. In addition to a regularization term intended to prevent class collapse, the method reported 95.90 ± 4.31 UAR on IEMOCAP under a LOSO speaker-independent speaker branch and 88.31 ± 5.60 on Att-HACK. The same paper also notes that the very high IEMOCAP UAR warrants scrutiny across different splits and protocols (Moine et al., 2021).

Transfer learning has been used in two distinct ways: from other speech tasks and from image recognition. A speaker-recognition-to-SER transfer pipeline reuses a ResNet34 and statistics pooling layer pretrained on VoxCeleb, replaces the post-pooling fully connected layers, and fine-tunes the new head for SER. On full IEMOCAP under leave-one-session-out evaluation, this system achieved UA 61.61% and WA 66.02% in the angry–happy–neutral–sad setup, UA 65.56% and WA 65.62% in the angry–excited–neutral–sad setup, and UA 64.14% and WA 63.61% when happy and excited were merged; the paper attributes the largest single ablation gain to statistics pooling (Padi et al., 2021). Cross-domain transfer from vision is exemplified by ResNet34 on spectrogram “images”: on combined RAVDESS+SAVEE, transfer learning improved accuracy from 45.8% to 57.3%, and transfer learning plus augmentation raised it further to 66.7% with macro-F1 0.631 (Vu, 26 Aug 2025).

Self-supervised learning has since become a major axis of SER. Frozen Wav2Vec 2.0 and HuBERT encoders with mean pooling and small classifiers were evaluated across RAVDESS, SAVEE, Emo-DB, AESDD, and SHEMO, where Wav2Vec 2.0 reached 97.67 ± 1.2 WA and 98.02 ± 1.3 UA on RAVDESS, and 95.52 ± 2.1 WA and 91.21 ± 1.3 UA on SHEMO, while HuBERT was stronger on SAVEE at 91.66 ± 1.2 WA and 90.48 ± 1.1 UA (Jafarzadeh et al., 2024). On ShEMO, frozen XLSR-53 features followed by a custom CNN achieved validation accuracy above 80% and slightly improved weighted average recall and F1 over recent wav2vec-based baselines, illustrating the same pattern in Persian emotional speech (Tehrani et al., 2023).

The benchmark perspective is captured by EMO-SUPERB, which evaluates 16 self-supervised speech models as frozen encoders over six SER datasets and nine settings. XLS-R-1B achieved the best average macro-F1 of 0.38352, with WavLM at 0.38334 and HuBERT at 0.38331; the benchmark further reports that many models place high weight on shallow layers, and that using ChatGPT-generated relabels for natural-language annotations yields an average relative gain of 3.08% on MSP-Podcast (Wu et al., 2024). A related efficiency-oriented line uses self-distillation on wav2vec 2.0: the deepest exit reached 91.4% test UAR on DEMoS, while early exits at layers 8 and 10 retained 90.7% and 90.5% with reduced parameter counts, enabling explicit accuracy–efficiency trade-offs (Ren et al., 2022).

5. Multimodal fusion, ASR integration, and physiology-informed SER

Multimodal SER commonly augments acoustics with text, either from human transcripts or ASR. A speech-and-text system built from HuBERT audio features, BERT text embeddings, and Whisper transcripts reported WA 79.89 and UA 77.68 on a seven-class corpus formed by combining CREMA-D, SAVEE, TESS, IEMOCAP, and RAVDESS, outperforming both classical feature-based baselines and HuBERT-only speech models (Sharma, 2023). This supports the repeated observation that text carries complementary emotion cues, especially when lexical polarity or intensifiers matter.

ASR-integrated SER extends that idea to realistic deployment. A thesis-length study shows that multimodal approaches consistently outperform unimodal ones, that Wav2Vec 2.0 middle layers are more emotion-relevant than final ASR-specialized layers, and that hierarchical co-attention over ASR hidden states, ASR text, and paralinguistic features reaches 68.4% UA on IEMOCAP, approaching the corresponding ground-truth-text baseline. The same work analyzes how WER affects text-only and bimodal SER, develops LLM-based and sequence-to-sequence ASR error correction, and reports that corrected transcripts combined with dynamic audio–text gating can raise IEMOCAP four-class accuracy to 76.66%, compared with 74.66% using the best raw ASR transcript condition (Li, 25 Jan 2026).

A different extension uses production physiology. The STEM-E2VA work introduces a Mandarin emotional dataset with parallel audio, electroglottography, and electromagnetic articulography, and fuses audio features with EGG-derived excitation features and EMA-derived articulatory statistics. Audio-only reached 79.97% accuracy, speech plus articulatory features reached 87.23%, and tri-modal speech plus excitation plus articulatory fusion reached 88.42%. When physiological signals were estimated from speech rather than measured, the same fusion strategy still improved on audio-only, reaching 82.69% (Zhang et al., 11 Nov 2025).

PhysioSER generalizes this physiological perspective within modern SSL pipelines. It constructs a vocal anatomy and physiology informed quartet of features—log-magnitude, log-magnitude rate, instantaneous frequency, and group delay—embeds them in a quaternion field, processes them with Hamilton-structured quaternion convolutions, aligns the resulting vocal representation with a frozen SSL branch through symmetric InfoNCE, and fuses both streams with a shallow Transformer. Across 14 datasets, 10 languages, and 6 backbones, it reports consistent gains in WA, UA, and macro-F1; for example, with frozen WavLM on CREMA-D, WA improved from 69.69 to 75.20, UA from 70.22 to 75.62, and F1 from 69.79 to 75.63, while training only about 2.03M parameters. The framework was also deployed in real time on the humanoid robot Ameca (Zhang et al., 3 Feb 2026).

6. Benchmarks, metrics, and reproducibility controversies

SER evaluation is dominated by weighted accuracy, unweighted accuracy, unweighted average recall, macro-F1, and confusion-matrix analysis. One common definition is

$WA = \frac{\sum_i n_i \cdot Acc_i}{\sum_i n_i},$

with $Acc_i$ the accuracy on class $i$ and $n_i$ the number of test samples in class $i$, while

$UA = \frac{1}{C}\sum_{i=1}^{C} Acc_i,$

where $C$ is the number of classes (Wang et al., 2019). UAR is the corresponding macro-average recall used in several speaker-independent studies (Moine et al., 2021). Macro-F1 is emphasized when label distributions are imbalanced or soft-label settings are used (Wu et al., 2024).

A recurrent misconception is that headline accuracy alone implies robust generalization. Some of the highest reported numbers come from controlled, acted, or speaker-dependent settings. The TESS MFCC-LSTM study used two professional female actors and an 80/20 split in which the same speakers appear in both train and test, and explicitly notes that such evaluation can inflate performance relative to spontaneous, noisy, multi-speaker corpora (Oluwademilade et al., 16 Apr 2026). The DRCNN work applied augmentation before splitting the data, so augmented samples appeared in train, validation, and test sets; it also used random, non-speaker-independent partitions (Niu et al., 2017). The CNN–Transformer work on IEMOCAP used a random 80/20 split rather than standard speaker-independent cross-session protocols, and the authors note that comparability to all published numbers is therefore limited (Tang et al., 2024).

EMO-SUPERB elevates this from anecdote to benchmark critique. It reports that 80.77% of SER papers yield results that cannot be reproduced, identifies leakage-prone dyadic partitioning as a systematic confound, and quantifies that studies with leakage report 4.011% improvements on average relative to leakage-free settings (Wu et al., 2024). This suggests that protocol specification—speaker independence, cross-validation design, handling of overlap in dyadic corpora, augmentation order, and train/dev/test partitioning—is a first-order scientific variable in SER rather than a minor implementation detail.

Another issue is cross-corpus robustness. Many strong within-corpus results are not accompanied by cross-corpus evaluation. Papers that explicitly note this limitation include transfer-learning and augmentation work on RAVDESS+SAVEE, self-supervised models on five corpora with random utterance-level splits, and several high-accuracy MFCC-based systems on TESS, SAVEE, or RAVDESS (Vu, 26 Aug 2025, Jafarzadeh et al., 2024, Lee et al., 4 Jul 2025). The benchmark literature therefore increasingly treats speaker-independent partitions, multi-dataset evaluation, and external-noise or real-world tests as necessary complements to within-corpus accuracy.

7. Applications, limitations, and emerging directions

Applications described across the literature include human–computer interaction, virtual assistants, conversational agents, customer service, call center analytics, market research, mental health and well-being monitoring, digital healthcare, social robotics, and humanoid robotic interaction (Oluwademilade et al., 16 Apr 2026, Jafarzadeh et al., 2024, Nigar, 2024, Zhang et al., 3 Feb 2026). Some systems have already been tested in applied settings: one SSL-based model was used on call-center conversations, and PhysioSER was validated in real-time deployment on a humanoid robotic platform (Jafarzadeh et al., 2024, Zhang et al., 3 Feb 2026).

The main limitations are equally consistent. Many benchmark corpora are acted, lexically repetitive, and small; speaker overlap across splits is common; class imbalance can distort global accuracy; and emotion categories themselves have fuzzy boundaries. ASR-integrated systems must contend with transcription errors, while physiology-informed methods must either collect specialized sensor data or rely on imperfect inversion from speech (Li, 25 Jan 2026, Zhang et al., 11 Nov 2025). Even when raw-waveform SSL encoders are strong, mean pooling may underuse frame-level dynamics, and highly expressive but rare cues such as laughter, sighs, or abrupt voice-quality changes may still be weakly modeled (Jafarzadeh et al., 2024, Wu et al., 2024).

Current research directions therefore converge on a few themes. One is richer representation learning: multi-resolution acoustics, channel and temporal attention, speaker-conditioned attention, self-supervised encoders, and physiology-informed amplitude–phase coupling all attempt to preserve information discarded by simpler pipelines (Wang et al., 2019, Moine et al., 2021, Zhang et al., 3 Feb 2026). Another is robustness under low-resource or noisy conditions, using transfer learning, augmentation, self-distillation, ASR error correction, and semi-supervised pseudo-labeling (Vu, 26 Aug 2025, Ren et al., 2022, Li, 25 Jan 2026). A third is methodological standardization, exemplified by EMO-SUPERB and by repeated calls for speaker-independent and cross-corpus evaluation (Wu et al., 2024). Taken together, these trajectories suggest that SER is increasingly understood not as a single-model classification problem, but as a multi-source inference problem in which acoustic structure, linguistic content, speaker identity, and speech-production physiology must be modeled under strict experimental control.