UniBind: Unified Multi-Modal Learning

• UniBind is a unified multi-modal representation framework that leverages LLM-generated text clouds to create a balanced, modality-agnostic embedding space across seven data types.
• It constructs a rich knowledge base by indexing category-level and instance-level multi-modal descriptions using frozen CLIP-style encoders for precise semantic alignment.
• UniBind significantly enhances recognition and retrieval performance across 14 benchmarks while reducing trainable parameters by up to 90% compared to image-centric approaches.

UniBind is a LLM-augmented framework for unified and balanced multi-modal representation learning across seven diverse data modalities: images, text, audio, point cloud, thermal, video, and event data. It addresses the limitations of prior CLIP-style and image-centered binding methods, such as ImageBind, which produce embedding spaces biased toward images and inadequately capture inter-modality semantics. UniBind introduces a modality-agnostic alignment center, utilizing clouds of text embeddings generated by LLMs and multi-modal LLMs to create a high-fidelity, semantically rich, and balanced embedding space shared across modalities. The result is a system that can be flexibly integrated into existing CLIP-style architectures, significantly boosting recognition and retrieval performance, while requiring substantially fewer trainable parameters (Lyu et al., 2024).

1. Motivation and Theoretical Basis

Conventional multi-modal representation methods, particularly variants of CLIP and ImageBind [Girdhar et al. 2023], use RGB images as a fixed alignment center, requiring all modalities to align against them via contrastive learning. This "image-centric" paradigm leverages established image–text datasets but induces bias, diminishing the representational quality for non-image modalities. Additionally, standard category-name prompts (e.g., "A photo of a [class]") used as text centers are insufficient for capturing the deep semantics inherent in multi-modal data, leading to sub-optimal modality mixing and inter-class discrimination. UniBind addresses these issues by decoupling the alignment center from any specific modality and employing rich, LLM-generated text clouds as class-wise anchors, thereby ensuring balanced modality representation and enhanced semantic grounding [(Lyu et al., 2024), Fig 1].

2. Knowledge Base Construction

UniBind constructs a comprehensive text-embedding knowledge base for each dataset, consisting of two key components:

• Category-level Descriptions: For each class $C_i$, one or more LLMs (such as GPT-4 or LLaMA) are prompted to generate up to 1,000 paraphrases or descriptive sentences $T^1_{C_i}, T^2_{C_i}, \dots, T^n_{C_i}$ of length ≤77 tokens.
• Instance-level Multi-modal Descriptions: Each individual sample from every modality (e.g., an image $I_i$, audio clip $A_i$, etc.) is processed by a multi-modal LLM (e.g., BLIP-2, LLaMA-Adapter) to generate a concise, modality-aware textual description ($T_{I_i}, T_{P_i}, \dots$), which captures nuances like shape, thermal pattern, or event dynamics.

Every description is indexed in the knowledge base as $$\{\text{ID}, \text{Category}, \text{Description}, \text{Source}\}$$ (see Fig 5). This dual-sourcing of text—one from LLMs and another from multi-modal LLMs—enhances semantic diversity and fidelity compared to single-source or category-name-only approaches.

3. LLM-Augmented Class-wise Embedding Centers

UniBind abstracts away from a single-modal center by creating a "cloud" of top-$K$ text embeddings per class, serving as a modality-agnostic alignment center.

• All descriptions are encoded by a frozen CLIP-style text encoder $F^T$ into $z=F^T(T)\in\mathbb{R}^d$.
• For each class $C_j$, the system computes cosine similarities between all candidate descriptions and the class-prompt embedding $T^1_{C_i}, T^2_{C_i}, \dots, T^n_{C_i}$0.
• The top $T^1_{C_i}, T^2_{C_i}, \dots, T^n_{C_i}$1 (empirically $T^1_{C_i}, T^2_{C_i}, \dots, T^n_{C_i}$2) highest-scoring descriptions are selected:

$T^1_{C_i}, T^2_{C_i}, \dots, T^n_{C_i}$3

These embeddings, $T^1_{C_i}, T^2_{C_i}, \dots, T^n_{C_i}$4, constitute the class-wise anchor. This representation encodes considerably finer semantic distinctions than a prompt mean, which is demonstrated empirically by enhanced cluster separation (Fig 4a, Fig 10).

4. Unified LLM-Augmented Contrastive Learning

Each modality encoder, $T^1_{C_i}, T^2_{C_i}, \dots, T^n_{C_i}$5, is frozen and extended with a lightweight, trainable linear projection head $T^1_{C_i}, T^2_{C_i}, \dots, T^n_{C_i}$6. For a batch of modality-$T^1_{C_i}, T^2_{C_i}, \dots, T^n_{C_i}$7 samples $T^1_{C_i}, T^2_{C_i}, \dots, T^n_{C_i}$8 and their associated descriptions $T^1_{C_i}, T^2_{C_i}, \dots, T^n_{C_i}$9, positive pairs $I_i$0 are constructed, and negatives are drawn from other batch samples.

The contrastive objective for modality $I_i$1 is:

$I_i$2

where $I_i$3 and $I_i$4 is the temperature (typically 0.07). The total loss aggregates across all modalities:

$I_i$5

This ensures equidistant alignment of all modalities to the same text cloud anchors, preventing image-dominated representational spaces.

5. System Integration and Training Protocols

UniBind is compatible with CLIP-style architectures such as CLIP, ImageBind, E-CLIP, AudioCLIP, PointBind, and PointCLIP. Integration entails freezing all backbone encoders, appending each with a modality-specific projection head ($I_i$6), and leveraging the frozen text encoder for both knowledge-base indexing and loss computation.

Parameter Efficiency: Only $I_i$7 and minimal cloud selection layers are trained, reducing the trainable parameter count by approximately 90% compared to full-model fine-tuning ($I_i$810% are updated).

Protocols:

• Zero-shot: The projection heads are frozen; predictions use the maximal cosine similarity over the class text cloud:

$I_i$9

• Fine-tuning: Only $A_i$0 are updated via $A_i$1 for 10–20 epochs, using AdamW with learning rates $A_i$2–$A_i$3 and batch sizes 64–1024.

Training only the projection heads preserves pre-trained knowledge in the backbones and avoids catastrophic drift.

6. Empirical Performance and Ablations

UniBind demonstrates statistically significant improvements across seven modalities and 14 benchmarks, including ImageNet-1K, Places-365, Caltech-101, ESC-50, Urban-S, ModelNet-40, ShapeNet, LLVIP, RGB-T, MSR-VTT, UCF-101, N-Caltech-101, and N-ImageNet.

Setting	Mean Gain	Notable Result
Zero-Shot (top-1)	+6.36% vs. prior art	Table 2
Fine-Tuning (ImageNet)	+6.75% (e.g. on PointBind)	Table 2, Fig 3
Retrieval (Recall@20)	+17.96% (event–image)	Table 4

No additional modality-specific pre-training or architectural changes are required; gains result solely from the richer, cloud-based, modality-agnostic alignment strategy.

Ablation findings:

• LLM-augmented Contrastive Learning (LCL): Omitting LCL yields poorly mixed clusters; its inclusion improves retrieval Recall@20 by +1.55–+17.96%, depending on modality pair and architecture.
• Embedding Center Localization (ECL): Using a single prompt for alignment costs up to –8.28% top-1 accuracy (e.g., PointBind on N-Cal). ECL with $A_i$4 achieves sharper, better-separated modality clusters.
• Knowledge Base Mix: Combining both LLM and multi-modal LLM outputs yields better results than either alone. $A_i$5 offers optimal balance between coverage and noise.

The synergy of all three components—rich knowledge base, ECL, LCL—is critical for the observed improvements.

7. Significance and Context within Multi-Modal Representation Learning

UniBind's modality-agnostic, LLM-augmented binding offers a unified resolution to the trade-offs imposed by prior image-centered architectures. By leveraging large, diverse clouds of text descriptions, the framework more effectively captures category semantics and modality-specific nuances, thus promoting balanced cluster formation and improving both within-modality and cross-modal retrieval and recognition. Its compatibility with diverse existing architectures, substantial parameter savings, and strong empirical gains underscore its practical significance in the field of multi-modal learning (Lyu et al., 2024). The framework highlights the value of integrating modern LLM outputs with multi-modal systems and sets a new benchmark for balanced, scalable, and semantically grounded multi-modal representation.