Speech-Preserving Facial Expression Manipulation

• Speech-Preserving Facial Expression Manipulation (SPFEM) is a technique that enables independent control of facial expressions while maintaining precise speech-driven mouth movements in videos.
• It leverages disentangled representations and specialized loss functions, such as lip-sync and vertex losses, to ensure audio-visual synchronization and temporal coherence.
• Architectural paradigms include 3D face models and GAN-based latent manipulation, offering fine-grained emotion editing for high-fidelity digital avatars and facial reenactment.

Speech-Preserving Facial Expression Manipulation (SPFEM) encompasses a class of techniques for modifying facial expressions in talking videos such that the original speech-driven mouth movements are strictly maintained. Unlike generic facial animation or reenactment methods, SPFEM specifically addresses the problem of disentangling and independently controlling emotional expression and speech articulation—two factors that are highly coupled in naturalistic human behavior. This enables fine-grained, temporally coherent, and perceptually consistent manipulation of an actor's affect while guaranteeing audio-lip synchronization, a requirement for high-fidelity speech-driven character animation, affect editing in post-production, and expressive digital avatars.

1. Theoretical Foundations and Challenges

SPFEM arises from the necessity to independently modulate facial expressions—typically determined by emotion, affect, or stylization—without disturbing the spatiotemporal integrity of lip-speech coordination. Traditional approaches either manipulate entire facial action unit sets, which inadvertently distort phoneme-specific mouth shapes, or rely on global latent code mixing incapable of preserving fine-grained speech content.

The fundamental technical challenge is the intrinsic entanglement of speech- and emotion-driven orofacial configurations. The same phonetic content can be realized in multiple emotional styles, e.g., a “p” phoneme in a smile versus a frown, leading to nontrivial perturbations if the control signals are not explicitly decoupled (Chen et al., 8 Apr 2025). Naïve parameter swapping (3DMM, action units, GAN latent codes) often leads to “melting” artifacts, poor lip sync, or diluted expression transfer (Lu et al., 19 Jan 2026, Cai et al., 2024).

2. Architectural Paradigms

SPFEM systems predominantly follow one of two architectural paradigms:

Paradigm	Key Mechanism	Representative Works
3D Face Model–based (3DMM)	Explicit parametric decomp; pose, identity, expression	(Papantoniou et al., 2021, Sun et al., 2022)
GAN-based Latent Manipulation	Direct editing in generator's latent or semantic spaces	(Wang et al., 18 Mar 2025, Chen et al., 8 Apr 2025)

3DMM-based pipelines reconstruct expression, identity, and pose vectors (e.g., FLAME or BFM09 models), and update only the target coefficients (emotion, mouth shape) needed for editing; this allows localized expression transfer but is often constrained by the limited capacity to represent detailed and nuanced emotions, and their tight coupling with speech kinematics (Papantoniou et al., 2021, Sun et al., 2022). GAN-based approaches employ disentangled latent representations (via adversarial or contrastive learning) and leverage advanced generators (StyleGAN, U-Net) to synthesize high-resolution, emotionally edited renderings, with direct or indirect supervision on lip sync (Chen et al., 8 Apr 2025, Cai et al., 2024).

A notable advancement is the two-stage hybridization employed by recent frameworks, such as the Talking Head Facial Expression Manipulation (THFEM) approach, which leverages a first-stage SPFEM model to edit facial expression on a static frame, followed by audio-driven talking head generation to restore mouth-motion realism using adjacent frame priors for temporal coherence (Lu et al., 19 Jan 2026).

3. Disentangling Speech and Emotion Representations

Robust SPFEM requires explicit disentanglement of emotion- and speech-content subspaces.

• Contrastive Decoupled Representation Learning (CDRL) Framework: CDRL (Chen et al., 8 Apr 2025) introduces two core modules:
  • CCRL (Contrastive Content Representation Learning): Extracts a speech-content embedding invariant to emotion, by aligning cross-attended image–audio representations with shared spoken content but diverging emotions, and repelling representations sharing emotion but differing in speech.
  • CERL (Contrastive Emotion Representation Learning): Extracts emotion embedding invariant to speech by correlating images with CLIP-derived emotion priors, enforcing content-independence.
  • These embeddings are frozen and used as strong regularization terms during the generator’s training, directly penalizing deviations in mouth shape (via content code) and emotional fidelity (via emotion code).
• Parametric and Keypoint-Based Decoupling: In PC-Talk (Wang et al., 18 Mar 2025), implicit keypoint representations decouple lip-speech deformation (learned via audio-aligned predictors and SyncNet losses) from emotion-specific non-speech deformations. Explicit subtraction of the neutral emotion's keypoint deformation isolates pure emotional displacement, enabling arbitrary intensity and region-wise mixing without compromising audio-lip alignment.
• Inter-Reconstructed Feature Disentanglement (IRFD): SPEAK (Cai et al., 2024) applies inter-reconstruction losses across triples of static (identity), dynamic (emotion), and dynamic (pose) encoded latents, ensuring each latent factor can be independently swapped and recomposed with negligible interference.

4. Speech Preservation Constraints and Loss Functions

Ensuring tight audio-lip synchronization is paramount.

• Lip-Sync Losses: Most state-of-the-art SPFEM systems employ explicit audio-visual synchronization losses, often using pre-trained SyncNet networks to align the distribution of mouth-region features with the corresponding audio segment (Wang et al., 18 Mar 2025, Cai et al., 2024). The losses may be contrastive (hinge, cosine), or based on direct distance between predicted and ground-truth mouth embeddings.
• Mouth Shape/Vertex Losses: L1 or L2 losses between predicted and true 3D mouth keypoints or mesh vertices reinforce the exact geometric opening and closure dynamics required for speech (Chen et al., 2023, Sun et al., 2022, Bozkurt, 2023).
• Differential/Velocity Regularization: Penalizing rapid or inconsistent changes in mouth kinematics across consecutive frames, often via velocity or frame-difference losses in the mesh or latent space, enforces temporal stability (Bozkurt, 2023, Sun et al., 2022).
• Correlation-Based Jaw Articulation Loss: Approaches such as Neural Emotion Director (Papantoniou et al., 2021) maximize the temporal correlation (Pearson’s ρ) between the jaw-parameter trajectory of original and edited sequences, allowing expressive variation while demanding that speech-driven fluctuations remain strictly preserved.
• Adversarial, Perceptual, and Cycle Consistency Losses: GAN-based photorealism and expression/identity preservation are ensured through multi-term adversarial losses (patch-based and full-image), VGG-based perceptual metrics, and cycle-consistency on expression parameters (Lu et al., 19 Jan 2026, Chen et al., 8 Apr 2025, Sun et al., 2022, Papantoniou et al., 2021).

5. Expression Manipulation Mechanisms and Controls

State-of-the-art SPFEM techniques provide expressive, high-fidelity, and fine-grained expression control:

• Continuous Emotion Editing: Recent works allow continuous modulation of expression intensity and even composition of multiple emotions across spatial regions of the face (e.g., mixing "angry eyes" with a "smile") (Wang et al., 18 Mar 2025, Sun et al., 2022). Editing is performed either via direct arithmetic in the latent/parameter space or by region-wise keypoint interpolation.
• Emotion-Conditional Latent Modulation: Adaptive normalization or latent code injection, modulated by emotion embedding, enables continuous and smoothly interpolated expression editing on the synthesized frames (Cai et al., 2024, Bozkurt, 2023).
• Sequence-Level and Per-frame Consistency: Adjacent-frame learning (Lu et al., 19 Jan 2026) explicitly penalizes inconsistencies across a window of consecutive frames, improving temporal coherence and realism during dynamic facial animation.
• User and Animator Controls: Architectures support intensity scaling, keyframe-level dials, and region-level blending through explicit conditioning vectors or GUI-driven sliders, allowing granular user control over affect without violating lip synchronization (Chen et al., 2023, Sun et al., 2022).

6. Evaluation Methodologies and Empirical Findings

Standardized metrics and subjective studies are now established for SPFEM assessment:

• Lip-Sync:
  • Lip Sync Error–Distance (LSE-D; lower is better), SyncNet Confidence (SYNC_C; higher is better), and max/mean vertex error (mm) are widely reported (Wang et al., 18 Mar 2025, Chen et al., 8 Apr 2025, Chen et al., 2023, Cai et al., 2024).
  • State-of-the-art frameworks such as PC-Talk and SPEAK consistently demonstrate lowest LSE-D and highest SYNC_C across MEAD and HDTF datasets, outperforming prior art in both neutral and emotional conditions (Wang et al., 18 Mar 2025, Cai et al., 2024).
• Emotional Expression Fidelity:
  • Cosine similarity of expression features (CSIM) and emotion classification accuracy by pre-trained emotion-recognition networks are used for objective assessment (Chen et al., 8 Apr 2025, Wang et al., 18 Mar 2025, Sun et al., 2022).
  • CDRL augmentation yields significant improvements in CSIM (e.g., NED baseline: 0.831→0.914 intra-ID on MEAD) (Chen et al., 8 Apr 2025).
• Visual Realism and Identity:
  • FID (Fréchet Inception Distance) and FAD (Fréchet ArcFace Distance) quantify image- and identity-realism, respectively (Lu et al., 19 Jan 2026, Chen et al., 8 Apr 2025).
  • Ablation studies confirm additive improvements from CDRL modules, keypoint-based editing, and adjacent-frame augmentations.
• Subjective Metrics:
  • User studies (n=20–50) assess realism, perceived emotion, and mouth-shape similarity, consistently favoring modern SPFEM frameworks over conventional talking head or facial reenactment pipelines (Wang et al., 18 Mar 2025, Lu et al., 19 Jan 2026, Papantoniou et al., 2021).

7. Research Trends, Limitations, and Open Problems

Current SPFEM research demonstrates rapid convergence to architectures that combine explicit representation disentanglement, strong synchronization constraints, and flexible expression conditioning. The integration of contrastive learning and adjacent-frame priors further enhances the fine-grained control and temporal consistency required for high-use scenarios.

However, persistent limitations remain:

• Residual Coupling: No approach achieves perfect decoupling of speech and emotion. Subtle artifacts can arise when emotional editing modulates regions (e.g., cheeks, lip corners) that interact with speech articulation (Lu et al., 19 Jan 2026, Chen et al., 8 Apr 2025).
• Resource Demands: Two-stage hybrid pipelines (SPFEM + AD-THG) incur high computational and parameter overhead, and artifacts from an erroneous first-stage edit can propagate uncorrected to the final synthesis (Lu et al., 19 Jan 2026).
• Dependence on Data Quality: Systems are sensitive to the curation of emotion priors and may generalize poorly to ambiguous or unseen blends of content and affect (Chen et al., 8 Apr 2025, Sun et al., 2022).

A plausible implication is that future SPFEM architectures will prioritize unified, end-to-end training schemes where cross-modal attention and multi-window synchronization are directly optimized, possibly integrating audio, visual, and semantic priors in a single transformer-based backbone (Lu et al., 19 Jan 2026). Scalability to high resolutions, generalization across diverse cultures and physiologies, and real-time performance remain active open areas.