Emotion-Audio-Guided Spatial Attention

• Emotion-audio-guided spatial attention involves neural systems amplifying emotion-rich zones in audio for effective emotion recognition.
• MFHCA leverages log-Mel spectrograms and hierarchical patterns to boost accuracy in speech emotion recognition through advanced attention mechanisms.
• VAANet combines visual and auditory features using sequential attention layers, enhancing emotion recognition performance across diverse modalities.

Emotion-audio-guided spatial attention refers to neural frameworks that employ attention mechanisms—typically spatial, temporal, and hierarchical—to selectively emphasize emotion-salient regions within audio-derived or multimodal representations for emotion recognition tasks. The paradigm is exemplified by recent advances in speech emotion recognition and video emotion recognition, notably the MFHCA architecture (Jiao et al., 2024) for auditory modalities and VAANet (Zhao et al., 2020) for audiovisual contexts. These methods allocate computational focus to regions or time segments most predictive of emotional state, often leveraging spectrogram and learned feature representations.

1. Multi-Spatial Fusion (MF) Module (MFHCA)

MF operates exclusively on log-Mel spectrogram inputs $X \in \mathbb{R}^{C \times H \times W}$, where $C$ channels span the Mel bands, $H$ denotes frequency bins, and $W$ covers time frames. The procedure begins with parallel convolutions emphasizing distinct axes:

• Temporal: $X_t = \mathrm{Conv}_{(k_h=10,k_w=2)}(X)$
• Frequency: $X_f = \mathrm{Conv}_{(k_h=2,k_w=8)}(X)$

The results are concatenated and processed through serial Global Receptive Field (GRF) blocks. Each GRF block produces output $Y = X + Y_a + Y_b$, combining a spatial attention branch and a large-context residual branch.

Spatial attention branch ($Y_a$) employs “strip” pooling for context aggregation along axes: $$z_{c}(h)=\frac{1}{W}\sum_{i=1}^{W}x_{c}(h,i) \qquad z_{c}(w)=\frac{1}{H}\sum_{j=1}^{H}x_{c}(j,w)$$ These pooled vectors are mixed, activated with Swish, and projected into axis-specific sigmoid-gated attention masks. The reweighting operation: $Y_{a}(i,j) = x(i,j) \odot g_{h}(i) \odot g_{w}(j)$ focuses the network’s response on emotion-rich time-frequency zones.

Large-context residual branch ($Y_b$) applies: $$Y_b = X + \mathrm{Up}(W_{3\times3}(\mathrm{AvgPool}_r(X)))$$ capturing broad contextual dependencies through downsampling, convolution, and upsampling.

After $N$ GRF blocks and optional projection, the feature map is reshaped to $f_{\rm spec} \in \mathbb{R}^{T \times d}$ for subsequent processing.

2. Hierarchical Cooperative Attention (HCA) Module (MFHCA)

The HCA module fuses spectrogram features $f_{\rm spec}$ with self-supervised Hubert representations $f_{\rm hubert}$ and their BiLSTM-encoded variant $f'_{\rm hubert}$. The fusion mechanism proceeds as follows:

• Compute cross-modal attention scores: $A = \mathrm{softmax}(f_{\rm spec} \, f'_{\rm hubert}^{T}) \in \mathbb{R}^{T \times L}$
• Use scores to attend to Hubert’s raw features: $$f_{\rm att}' = A \times f_{\rm hubert} \in \mathbb{R}^{T \times d'}$$
• Fuse attended audio context into spectrogram features: $$f_{\rm out} = [f_{\rm spec} \Vert f_{\rm att}'] \in \mathbb{R}^{T \times (d+d')}$$ No additional gating or residuals are added, with hierarchical fusion solely from attention over progressively contextualized audio embeddings.

3. Spatial and Channel-wise Attention in Multimodal Networks (VAANet)

VAANet implements sequential attention layers in both visual and audio streams. For video, spatial attention for each segment feature $F_i \in \mathbb{R}^{m \times n}$ is computed: $$H_i^S = W^{S_1}(W^{S_2}F_i^{\top})^{\top} \in \mathbb{R}^{m \times 1}$$

$A_i^S = \mathrm{Softmax}(H_i^S)$

$F_i^S = A_i^S \otimes F_i$

After spatial pooling, channel-wise attention is computed similarly. Temporal attention aggregates segment-level features for both modalities: $$H^T = W^{T_1}(W^{T_2}P^{\top})^{\top} \qquad A^T = \mathrm{ReLU}(H^T)$$

$E^V = \sum_{i=1}^t p_i A_i^T$

Fusion concatenates attended visual and audio vectors before classification.

There is no direct audio-to-spatial fusion for visual attention; attention masks are derived from unimodal representations in the published VAANet instantiation.

4. Training Objectives and Loss Formulations

MFHCA utilizes standard cross-entropy over final predictions: $$\mathcal{L}_{\rm CE} = -\frac{1}{B}\sum_{b=1}^{B}\sum_{c=1}^{C} y_{b,c}\,\log(\hat p_{b,c})$$ There are no auxiliary or attention-specific supervisory losses.

VAANet introduces the polarity-consistent cross-entropy: $\mathcal{L}_{PCCE} = -\frac{1}{N}\sum_{i=1}^N\sum_{c=1}^C (1 + \lambda g(c, y_i)) \mathds{1}_{[c = y_i]}\,\log p_{i,c}$ where $g(c, y)$ indicates class polarity opposition and $\lambda$ is a penalty.

5. Empirical Performance of Emotion-Audio-Guided Spatial Attention

MFHCA achieves significant improvements over previous audio-only models on IEMOCAP (4-class), reporting weighted accuracy (WA) of $74.24\%$ (a $2.6\%$ increase) and unweighted accuracy (UA) of $74.57\%$ (up $1.87\%$). Ablation studies indicate:

Configuration	WA	UA
Spec alone	62.13	62.25
Spec + MF	73.72	74.53
Spec + HCA	73.19	73.72
Full (MF + HCA)	74.24	74.57

MF contributes approximately $1.6\%$ gain in WA over spec+Hubert, with HCA providing an additional $0.5\%$ improvement. t-SNE projections corroborate enhanced class separability with HCA.

In VAANet, experimental results on VideoEmotion-8 and Ekman-6 verify that combined spatial, channel, and temporal attention modules in visual and audio streams yield improved accuracy over previous pipelines, with explicit polarity penalties guiding more discriminative attention allocation (Zhao et al., 2020).

6. Architectural and Training Hyperparameters

MFHCA’s MF module processes log-Mel spectrograms with parallel convolutions and stacked GRF blocks, while HCA utilizes both spectrogram and Hubert features and a three-layer classifier. VAANet’s backbone comprises 3D ResNet-101 (visual, pre-trained on Kinetics) and 2D ResNet-18 (audio, pre-trained on ImageNet), with temporal segmentation ($t=10$), $k=16$ frames per segment, and input resolution $112 \times 112$. Batch size is $32$, trained for $150$ epochs using Adam optimizer ($2 \times 10^{-4}$ learning rate), and visual stream data augmentation includes random crop and horizontal flip. Attention modules are implemented via learned linear layers or $1 \times 1$ convolutions.

7. Significance and Implications

Emotion-audio-guided spatial attention frameworks, as instantiated in MFHCA and VAANet, enable models to selectively amplify segments and regions most relevant to emotion inference, yielding quantifiable improvements in accuracy. MFHCA demonstrates that multi-spatial fusion via axis-aware masks, combined with hierarchical audio feature integration, is effective for speech emotion recognition. VAANet extends these principles to the visual domain through modular attention schemes. A plausible implication is that further development of cross-modal attention and spatial-temporal alignment strategies will continue to enhance emotion recognition systems, particularly in complex multimodal environments (Jiao et al., 2024, Zhao et al., 2020).