Multimodal Deep Learning for AVSR

• AVSR is defined as the integration of audio and visual cues that dynamically adjust modality weighting to improve speech recognition under adverse conditions.
• Key methods include sequence-to-sequence attention, adversarial fusion for modality invariance, and transformer-based cross-modal interactions to optimize performance.
• State-of-the-art systems leverage end-to-end training, parameter-efficient adaptations, and temporal modeling to achieve reduced error rates across variable noise levels.

Multimodal deep learning architectures for audio-visual speech recognition (AVSR) integrate complementary audio and visual streams to enhance speech recognition robustness, especially under adverse acoustic conditions. Advances in deep learning have enabled the design of highly expressive models capable of learning complex cross-modal representations, adaptive attention, and temporally synchronized features. Recent architectures target end-to-end fusion, modality-invariance, dynamic adaptation to signal quality, and parameter-efficient scaling to large datasets. Below, key architectural approaches are surveyed, illustrating the spectrum of fusion mechanisms, sequence modeling strategies, and their empirical performance in benchmark noisy and clean settings.

1. Sequence-to-Sequence and Attention-Driven Fusion Paradigms

Early deep AVSR systems often leveraged parallel encoders for audio and visual modalities, followed by a decoder that emits the output sequence conditioned on fused cross-modal representations. In sequence-to-sequence AVSR frameworks, the encoder-decoder paradigm is extended to learn not only temporal alignments but also the relative importance of modalities at each generation step. Modal attention mechanisms compute learned soft weights over modality context vectors for every decoding step as follows (Zhou et al., 2018):

• Audio encoder: multi-layer bidirectional LSTM downsampling acoustic features (e.g., 71-dim Mel-filterbanks from 100Hz→25Hz).
• Video encoder: CNN stack (processing raw RGB lip crops) followed by BLSTM layers for visual temporal modeling.
• Decoder: LSTM receives fused context computed via content-based attention over each encoder and a trainable "modality attention" softmax:

$$[\alpha_i^a;\,\alpha_i^v] = \operatorname{softmax}([z_i^a;\,z_i^v]),\qquad c_i = \alpha_i^a c_i^a + \alpha_i^v c_i^v$$

where $z_i^m$ is a modality-specific attention score from the decoder state and corresponding context vector.

This approach allows dynamic reweighting across modalities based on current input reliability, demonstrated to yield relative CER improvements of 2–36% compared to audio-only models as SNR deteriorates (Zhou et al., 2018). Unlike simple concatenation or late fusion architectures, modality attention provides explicit adaptability to noise and can generalize to other multimodal tasks with correlated information.

2. Modality-Invariant Representation Learning and Adversarial Fusion

A key challenge in deep multimodal learning is the distributional gap between audio and visual representations, complicating effective fusion. Adversarial architectures such as MIR-GAN introduce a generator–discriminator paradigm: modality-specific transformers encode each stream, a generator produces a fused, frame-level modality-invariant embedding, and a discriminator is trained to distinguish whether a representation is audio, visual, or fused (Hu et al., 2023):

• Modality-specific streams ($f_a^\text{spe}$, $f_v^\text{spe}$ via transformer encoders).
• MIR generator performs hybrid-modal cross-attention followed by a learned gating mask to generate $f_{va}^\text{inv}$—a representation maximally ambiguous as to its origin (audio vs visual).
• Discriminator is optimized adversarially so the fused stream approaches output $p=0.5$.
• Training objective:

  $$L_{\text{total}} = L_{\text{rec}} + \lambda_{\text{adv}}L_{\text{adv}} + \lambda_{\text{MIM}}L_{\text{MIM}}$$

  where $L_{\text{MIM}}$ is a mutual information maximization term enforcing foreground semantic alignment.

Concatenating invariant and specific streams as input to the final encoder-decoder achieves new state of the art (2.1%/8.5% WER on LRS3 clean/noisy), and ablations confirm the criticality of the adversarial and mutual information components for robust fusion (Hu et al., 2023).

3. Temporal Modeling and Modal-Temporal Fusion with Recurrent Architectures

Temporal dynamics are central to AVSR: both audio and visual cues evolve over time, and effective models must jointly capture intra-modal dependencies and cross-modal synergy. The auxiliary multimodal LSTM (am-LSTM) architecture implements parallel LSTM streams for each modality, fuses them via learned additive projections at every time step, and attaches auxiliary classifiers to each stream (Tian et al., 2017):

• Video/audio features are extracted and reduced (e.g., VGG-16 and PCA for vision; spectrogram and PCA for audio).
• Each modality forwards its sequence into an LSTM; hidden states are fused by $$f_t = \tanh(\frac{2}{3}(h_t^\text{v} + h_t^\text{a}))$$.
• Both main and auxiliary MLP classifiers receive temporally pooled hidden states.
• The total loss includes squared multi-label margin terms for main and auxiliary heads.

am-LSTM achieves competitive accuracy (up to 89.1% on AVLetters2), converges swiftly due to auxiliary supervision, and can be extended to more advanced fusion mechanisms or bidirectional encoders (Tian et al., 2017).

4. End-to-End Training and CTC for Sequence Alignment

End-to-end AVSR systems trained with connectionist temporal classification (CTC) loss function address the alignment issue between input frames and output phoneme/viseme sequences (Sanabria et al., 2016). Feature-level fusion is performed by direct concatenation prior to the RNN (BLSTM) backbone, which then emits frame-wise label posteriors:

• Audio: MFCC/FBank features at 33ms frames, ±1 context stacking.
• Video: landmark and SIFT descriptors aligned at the same frame rate.
• Four-layer BLSTM, softmax over K+1 labels (phoneme + blank).
• CTC collapses all valid alignment paths for the label sequence, enabling alignment-free training.

CTC “peaky” outputs serve as implicit alignments, and clean/noisy evaluations on large-vocabulary datasets demonstrate that fused models outperform audio-only and video-only across all SNRs, especially under noise (Sanabria et al., 2016). This confirms the principle that deep sequence models plus appropriate fusion can learn modal reliability without explicit synchronization mechanisms.

5. Transformer, Attention, and Cross-Modal Interaction at Scale

Recent transformer-based multimodal AVSR architectures have introduced deeper fusion, cross-modal interaction, and multi-layer integration to further close the performance gap. Notable examples include:

• Multi-layer cross-attention (MLCA): Cross-attention modules are interleaved at multiple depths in audio and visual encoder stacks such that each modality is contextually refined using the other, and intermediate CTC losses encourage discriminative fusion at every stage (Wang et al., 2024).
• Global interaction and local alignment (GILA): Stacks layers of intra-modal and cross-modal attention, introduces a bottleneck for joint refinement, and includes both within-layer and cross-layer contrastive losses to enforce frame-level and hierarchical alignment between modalities, yielding consistent WER reduction even without large unsupervised pre-training (Hu et al., 2023).

These designs demonstrate that explicit modeling of global and local cross-modal correlations (via attention, cross-attention, or contrastive objectives) enables robust multimodal representations and effective adaptation to varying SNR and dataset size.

6. Modality-Specific Versus Modality-Invariant Fusion and Adaptation

Hybrid AVSR models alternate between modality-invariant and modality-specific fusion, reflecting the complementary strengths of each approach. Reinforcement learning-based fusion frameworks (e.g., MSRL (Chen et al., 2022)) dynamically harmonize modality-invariant and modality-specific (vision) representations in the auto-regressive decoding process, guided by a reward function directly tied to task-specific WER. The policy network adaptively computes a convex combination coefficient for fusion at each decoding step, optimizing for both fidelity and resilience to modality degradation.

Similarly, self-supervised and adversarial approaches—such as MIR-GAN—use explicit regularization and adversarial training to prevent single-modality dominance, ensuring the system remains robust when one stream is corrupted (Hu et al., 2023).

7. Scaling to LLM-based and Modern Self-Supervised Architectures

With the advent of pre-trained multimodal encoders and LLMs, AVSR systems increasingly exploit frozen, high-capacity feature extractors and scalable fusion mechanisms. For instance, Llama-AVSR utilizes pre-trained audio and visual encoders to produce downsampled token streams, projects them into the LLM embedding space via modality-specific linear adapters, and concatenates them (along with a text prompt) to feed into a frozen LLM that is adapted via lightweight LoRA modules (Cappellazzo et al., 2024). This scheme allows efficient adaptation with as few as ~57M trainable parameters layered atop models with billions of frozen weights, establishing new SOTA WER (0.77% on LRS3 AVSR) and supporting efficient scaling via token compression.

Extensions such as Llama-SMoP generalize single-adapter fusion to a sparse Mixture-of-Experts (MoE) of projectors, allowing each modality to selectively activate modality-specific expert projectors, further enhancing WER and recognizing the differing statistical characteristics of audio and visual streams (Cappellazzo et al., 20 May 2025).

Matryoshka-based models (e.g., Llama-MTSK and Omni-AVSR (Cappellazzo et al., 9 Mar 2025, Cappellazzo et al., 10 Nov 2025)) introduce multi-granularity representation learning, such that a single model supports multiple rates of audio/video compression, catering to arbitrary computational budgets at inference and allowing elastic trade-off between accuracy and cost without retraining. These systems further extend parameter-efficient adaptation by mixing global and scale-specific LoRA modules and supporting unified training for ASR, VSR, and AVSR.

Table: Representative Architectures and Fusion Mechanisms

Architecture	Fusion Stage	Modality Adapter Type
Modality Attention (Zhou et al., 2018)	Decoder-side soft-attention	None / learnable scoring
MIR-GAN (Hu et al., 2023)	Concatenated invariant & specific	Adversarial+contrastive
am-LSTM (Tian et al., 2017)	Per-timestep LSTM fusion	Additive / simple MLP
MLCA-AVSR (Wang et al., 2024)	Multi-depth cross-attention	Transformer/Branchformer
Llama-AVSR (Cappellazzo et al., 2024)	Pre-prompt tokens, frozen LLM	Linear+LoRA
Llama-SMoP (Cappellazzo et al., 20 May 2025)	MoE projector module	Sparse expert selection
Omni-AVSR (Cappellazzo et al., 10 Nov 2025)	Joint token concatenation	Matryoshka LoRA*

*Editor's term: Matryoshka LoRA refers to nested, scale-adaptive low-rank adapters.

Summary and Outlook

Multimodal deep learning architectures for AVSR encompass a spectrum from early, feature-level fusion in end-to-end recurrent networks to contemporary approaches leveraging cross-modal attention, adversarial regularization, and scalable, parameter-efficient adaptation atop powerful self-supervised backbones and LLMs. Explicit modeling of cross-modal interactions, dynamic modality weighting, and the ability to operate robustly under severe noise and at variable computational budgets are hallmarks of state-of-the-art systems. Mechanisms such as matryoshka compression, mixture-of-expert projectors, and reinforcement-learned fusion are advancing the field towards models that deliver accuracy, adaptability, and efficiency in real-world, noisy, and resource-constrained settings (Zhou et al., 2018, Hu et al., 2023, Cappellazzo et al., 2024, Cappellazzo et al., 20 May 2025, Cappellazzo et al., 10 Nov 2025).