Minimal Gated Multimodal Unit (mGMU)

• mGMU is a lightweight fusion module that combines two modality-specific latent vectors using a single learned sigmoid gate to balance their contributions.
• It employs linear projections with tanh activations and a unified gate, thereby reducing parameter count and computational overhead compared to standard GMUs.
• Its effective integration in schizophrenia-spectrum assessments demonstrates improved weighted F1 and AUC-ROC scores while maintaining model compactness.

The Minimal Gated Multimodal Unit (mGMU) is a lightweight fusion module designed to combine two modality-specific latent vectors into a single joint representation in multimodal learning frameworks. It employs a single learned sigmoid gate to adaptively weight the contributions of each modality, thereby offering computational efficiency while preserving the adaptive fusion mechanism characteristic of fuller Gated Multimodal Units. The mGMU is particularly suited for settings that require frequent bimodal fusions where parameter count, computational cost, and network size are constraints, yet robust intermediary representations are necessary, as demonstrated in its application to schizophrenia spectrum assessment using audio, video, and text modalities (Premananth et al., 2024).

1. Mathematical Formulation

The mGMU accepts two input feature vectors—$x_1 \in \mathbb{R}^n$ from modality 1 and $x_2 \in \mathbb{R}^m$ from modality 2—and produces a fused latent vector $h \in \mathbb{R}^p$. The unit defines:

• $W_1 \in \mathbb{R}^{p \times n}$, $W_2 \in \mathbb{R}^{p \times m}$: modality-specific projection matrices,
• $b_1, b_2 \in \mathbb{R}^p$: bias terms for projections (often omitted in notation),
• $W_z \in \mathbb{R}^{k \times (n + m)}$ and $b_z \in \mathbb{R}^k$: parameters for gate projection (with $k = p$ by design),
• $\sigma(\cdot)$: element-wise sigmoid activation,
• $\tanh(\cdot)$: hyperbolic tangent nonlinearity.

The forward computations are:

$$\begin{align*} h_1 &= \tanh(W_1 x_1 + b_1) \ h_2 &= \tanh(W_2 x_2 + b_2) \ z &= \sigma(W_z [x_1; x_2] + b_z) \ h &= z \odot h_1 + z \odot h_2 \end{align*}$$

Here $[x_1; x_2]$ denotes concatenation and $\odot$ is element-wise multiplication. Each dimension of $z$ gates the corresponding dimensions of both modality projections in $h_1$ and $h_2$. When $z_i$ approaches one, the $i$-th dimension of both modalities is retained; near zero, it is suppressed for both.

2. Data Flow and Implementation

The mGMU operational pipeline consists of the following steps:

1. Input projection: Inputs $x_1$ and $x_2$ from unimodal encoders are linearly transformed and passed through $\tanh$, yielding latent vectors $h_1$ and $h_2$.
2. Gate computation: The raw inputs are concatenated and projected via $W_z$ (plus $b_z$), with $\sigma$ applied to produce gate vector $z$.
3. Fusion: The outputs $h_1$ and $h_2$ are combined via elementwise multiplication with $z$, producing the final multimodal representation $h$.

Pseudocode:

def mGMU_forward(x1, x2): a1 = W1 @ x1 + b1 # [p] a2 = W2 @ x2 + b2 # [p] h1 = tanh(a1) # [p] h2 = tanh(a2) # [p] concat = concatenate(x1, x2) # [n + m] gz = Wz @ concat + bz # [p] z = sigmoid(gz) # [p] h = z * h1 + z * h2 # [p] return h

3. Comparison with Standard Gated Multimodal Units

A key distinction between the mGMU and standard GMU (Arevalo et al., 2017) is the gating strategy:

Feature	Standard GMU	Minimal GMU
Gates	Two (or $z$ and $1\!-\!z$ for each modality)	Single vector $z$
Fusion formula	$h = z \odot h_1 + (1-z) \odot h_2$	$h = z \odot h_1 + z \odot h_2$
Parameter count	$W_1$, $W_2$, $W_z$, multiple biases	$W_1$, $W_2$, $W_z$, fewer
Computational overhead	Higher (additional multiplies, more gates)	Lower
Adaptive ability	High	Remains adaptive

By eliminating one gating projection and using a single gate identically on both modalities, the mGMU reduces parameter count and inference-time computation. Empirical results demonstrate comparable, and in some sensor-fusion and NLP tasks superior, performance versus more complex gating mechanisms at lower computational cost (Premananth et al., 2024).

4. Integration in Multimodal Assessment Frameworks

Within the referenced multimodal schizophrenia-spectrum diagnostic framework, the mGMU is integrated as follows:

• Unimodal encoding: Audio and video are processed via STS-CNN encoders, then aggregated to 128-dimensional temporal latent features via LSTM; text is embedded and similarly reduced by CNN + LSTM to 128 dimensions.
• Bimodal intermediate fusion: Each pair among audio, video, and text is fused using a dedicated mGMU, resulting in three joint representations ($h_{AV}$, $h_{AT}$, $h_{VT}$).
• Final fusion and classification: The three bimodal vectors are concatenated and fed to fully connected layers for 3-way classification (schizophrenia subtype or healthy control).

The complete model, containing three mGMUs, is reported at 897k trainable parameters, approximately 54% fewer than gated-attention baselines (1.93M), with superior classification accuracy (Premananth et al., 2024).

5. Hyperparameters and Training Configuration

The deployment of the mGMU in the schizophrenia-spectrum assessment task is characterized by:

• Optimizer: Adam with initial learning rate $1 \times 10^{-3}$,
• Learning-rate decay: Halved if no improvement in validation loss for 25 epochs,
• Early stopping: After 50 epochs without validation loss improvement,
• Epochs: Maximal cap of 300,
• Loss weighting: Per-class weights to address imbalance,
• Input segments: 40s for audio and 20s for video, both with 5s overlap,
• Training time: $\sim$12.5 minutes per run on an NVIDIA RTX 3090.

6. Empirical Results and Performance Impact

The use of the mGMU as the sole intermediate fusion mechanism produces decisive performance improvements:

• Without gating (concatenation): Weighted F1 $\approx$ 0.5538, AUC-ROC $\approx$ 0.7859,
• With mGMU intermediate fusion: Weighted F1 $\approx$ 0.6547, AUC-ROC $\approx$ 0.8214,
• Model compactness: The mGMU-enhanced network (897k parameters) is markedly smaller than gated-attention alternatives, yet achieves higher F1 and AUC-ROC.

Late fusion with mGMU also yields improvements over naive late fusion, though the gains are less pronounced at this stage (Premananth et al., 2024).

7. Significance and Practical Considerations

The mGMU's design—minimal gating and strong regularization of parameter count—addresses critical challenges in multimodal fusion where data volume, computational budget, or interpretability restrict the use of more elaborate attention- or gating-based fusion schemes. Its efficacy in real-world subject classification tasks, particularly in the domain of clinical assessment, underscores its utility. A plausible implication is that the mGMU architecture may generalize to other sensor-fusion or NLP tasks where reproducibility, parameter efficiency, and robust adaptive blending are required.

In summary, the Minimal Gated Multimodal Unit provides a parameter- and computation-efficient mechanism for data-driven multimodal fusion, as validated in a large-scale schizophrenia-spectrum assessment framework, delivering statistically significant gains in discriminative performance relative to both naive concatenation and more complex gated-attention models (Premananth et al., 2024).