Bidirectional Learning of Facial Action Units and Expressions via Structured Semantic Mapping across Heterogeneous Datasets

Abstract: Facial action unit (AU) detection and facial expression (FE) recognition can be jointly viewed as affective facial behavior tasks, representing fine-grained muscular activations and coarse-grained holistic affective states, respectively. Despite their inherent semantic correlation, existing studies predominantly focus on knowledge transfer from AUs to FEs, while bidirectional learning remains insufficiently explored. In practice, this challenge is further compounded by heterogeneous data conditions, where AU and FE datasets differ in annotation paradigms (frame-level vs.\ clip-level), label granularity, and data availability and diversity, hindering effective joint learning. To address these issues, we propose a Structured Semantic Mapping (SSM) framework for bidirectional AU--FE learning under different data domains and heterogeneous supervision. SSM consists of three key components: (1) a shared visual backbone that learns unified facial representations from dynamic AU and FE videos; (2) semantic mediation via a Textual Semantic Prototype (TSP) module, which constructs structured semantic prototypes from fixed textual descriptions augmented with learnable context prompts, serving as supervision signals and cross-task alignment anchors in a shared semantic space; and (3) a Dynamic Prior Mapping (DPM) module that incorporates prior knowledge derived from the Facial Action Coding System and learns a data-driven association matrix in a high-level feature space, enabling explicit and bidirectional knowledge transfer. Extensive experiments on popular AU detection and FE recognition benchmarks show that SSM achieves state-of-the-art performance on both tasks simultaneously, and demonstrate that holistic expression semantics can in turn enhance fine-grained AU learning even across heterogeneous datasets.

• The paper presents a Structured Semantic Mapping (SSM) framework that bidirectionally learns facial action units (AUs) and facial expressions (FEs) to address heterogeneous annotation challenges.
• It employs a CLIP-based visual encoder with Mixture-of-Experts and Textual Semantic Prototypes to dynamically mediate context-sensitive AU-FE associations.
• Extensive experiments reveal consistent improvements in F1 for AU detection and UAR/WAR for expression recognition, demonstrating the framework’s scalability and robustness.

Bidirectional Learning of Facial Action Units and Expressions via Structured Semantic Mapping Across Heterogeneous Datasets

Introduction and Motivation

The joint analysis of facial action units (AUs) and facial expressions (FEs) is fundamental in affective computing, particularly for fine-grained muscular activation recognition and holistic affective state assessment. While numerous studies have explored AU-to-FE knowledge transfer or rely on costly multi-label homogeneous datasets, the reciprocal, bidirectional learning of AUs and FEs under heterogeneous data annotation paradigms remains largely unaddressed. This paper presents a Structured Semantic Mapping (SSM) framework that systematically establishes a bidirectional adaptive knowledge transfer scheme using dynamic vision–language modeling, robustly bridging action unit and expression semantics across dataset and annotation disparities.

Problem Analysis and Joint-Learning Paradigm

Traditional multi-task learning methods leverage dataset homogeneity for bidirectional AU–FE gains but are bottlenecked by annotation cost and limited generalizability. Predominant approaches operate under unidirectional AU→FE transfer on heterogeneous datasets, suffering from mismatched annotation, semantic asymmetry, and pronounced domain biases due to laboratory-constrained AU distributions.

Figure 1: Paradigms of joint AU–FE learning, highlighting the SSM strategy of explicit AU$\leftrightarrow$FE modeling via semantic mediation, which outperforms both conventional MTL and previous unidirectional approaches under heterogeneous data.

This motivates techniques able to exploit only heterogeneous annotation regimes, exploiting scalability without constraining application to rare multi-label corpora. A core hypothesis validated in the paper is that stable bidirectional AU$\leftrightarrow$FE gains can be achieved with appropriately structured cross-task semantic mediation, even absent strict label alignment or identical collection domains.

Structured Semantic Mapping (SSM) Framework

The SSM framework integrates three pivotal components: a CLIP-based Mixture-of-Experts (MoE) visual backbone, a Textual Semantic Prototype (TSP) module, and a Dynamic Prior Mapping (DPM) module.

Visual Encoder: A shared CLIP ViT-B/16 visual encoder augmented with MoE layers yields discriminative representations for both DFER and AU-video segments, with task-adapted Transformer-based temporal modules.

Figure 2: The Baseline architecture, showcasing CLIP+MoE visual encoding with task-specific temporal modules, foundational to SSM's joint representation learning.

Textual Semantic Prototype (TSP): Semantic prototypes for AUs (from FACS atomic action descriptions) and FEs (constructed as compositional AU phrase sets per expression) are encoded via a frozen CLIP text encoder, facilitated by learnable prompt vectors for context adaptation.

Dynamic Prior Mapping (DPM): Cross-task semantic mediation is realized by bidirectional, FACS-initialized and data-optimized association matrices ($\mathbf{W}^{\text{au} \to \text{exp}}$, $\mathbf{W}^{\text{exp}\to\text{au}}$), which enable context-sensitive composition and decomposition of semantic prototypes. Through dynamically-weighted residual updates and contrastive supervision, SSM enforces adaptive, interpretable knowledge transfer between AU and FE tasks.

Figure 3: Overview of the SSM framework, detailing the dual-branch architecture and all semantic mediation dependencies.

Figure 4: Illustration of the DPM module, depicting bidirectional composition/decomposition operations on task semantics driven by learnable association matrices.

Experimental Results and Analysis

AU Detection

On BP4D and DISFA, SSM achieves superior or matching state-of-the-art F1, outperforming AU-focused, self-supervised, and vision–language baselines (e.g., FMAE-IAT, CausalAffect, MAE-Face), with especially robust improvements on context-dependent units (e.g., AU15/17/23).

Notable result: SSM's joint-training yields consistent F1 gains over its strong single-task baseline and prior vision–language fusion approaches (BP4D mean F1: 68.5 vs. 67.4 for next-best; DISFA mean F1: 71.9 vs. 70.8).

DFER

On DFEW, FERV39K, and MAFW, SSM consistently achieves or exceeds prior state-of-the-art UAR/WAR, outperforming CLIP-based and self-supervised models (DFEW UAR: 64.83, WAR: 75.37). Gains are accentuated on low-sample or compositionally complex emotion categories (as seen for 'disgust' and 'fear').

Figure 5: Data scaling study showing gains for both AU and FE tasks as the paired-task data increases, confirming effect robustness beyond sample size.

Ablations

Ablation studies confirm that each SSM component—semantic textual prototypes (TSP), dynamic prior mapping (DPM), and bidirectional learning—contributes non-redundant improvement, with DPM providing the most critical effect, especially when initialized with FACS priors. Linear or MLP shortcut mappings, or frozen DPM weights, lead to statistically significant performance drops.

Visualization and Semantic Structure Adaptation

Self-attention heatmaps illustrate that joint/semantic supervision induces more spatially organized and physiologically consistent attention, capturing multi-site muscular coordination.

Figure 6: Attention heatmaps reveal denser, muscle-group-structured focus under SSM, confirming enhanced representational richness for both AUs and FEs.

Moreover, visualization of learned DPM weights shows data-driven departure from initial priors and non-transposed, task-asymmetric mappings, reflecting real-world distribution adaptation and avoiding hard-coding errors.

Figure 7: SSM-learned bidirectional semantic mapping matrices, initial vs. random vs. prior-based, demonstrating convergence to context-sensitive associations.

Theoretical and Practical Implications

The SSM framework demonstrates that textual mediation enables effective cross-domain affective behavior modeling without requiring joint annotation or context overlap. The bidirectional, dynamically-learned mapping avoids the brittleness of static knowledge graphs or pseudo-label transfer, capturing real-world heterogeneity and annotator/subject variability.

Practically, this architecture immediately enables domain-extension and scalable deployment of affect recognition solutions, leveraging both robust laboratory AU data and unconstrained web-scale FE videos. The semantic-level adaptation provides a robust scaffold for future applications including cross-lingual, cross-cultural affect transfer, composite affect synthesis, and human-prior enhanced vision–language interfaces.

Limitations and Future Work

SSM's reliance on FACS-derived priors and prompt tuning does not fully resolve semantic granularity and label mapping mismatches in extreme cross-domain settings. Zero-shot results, though improved over prior art, indicate nontrivial domain shift remains. Current design abstracts away intra-sample interaction and multimodal cues: extending to dense spatio-temporal, hierarchical, and audio-visual reasoning is a clear next step.

Conclusion

This paper systematically establishes that adaptive, bidirectional AU$\leftrightarrow$FE learning is both theoretically beneficial and empirically realizable under heterogeneous annotation and domain settings. The SSM framework provides a general, extensible mechanism for semantic mediator–based affective behavior understanding, outperforming both unidirectional and rigid multi-task protocols. By decoupling annotation and label constraints via shared semantic spaces and dynamic prior adaptation, SSM points the way toward robust, scalable, and physiologically-informed affective computing strategies.