e5-omni: Cross-Modal Embedding Framework

• e5-omni is an omni-modal representation framework that maps diverse data types into a unified embedding space for robust cross-modal retrieval.
• It integrates modality-aware temperature calibration and hard negative curriculum learning to overcome modality-dependent noise and imbalance in contrastive tasks.
• The approach employs covariance-regularized batch whitening to align embedding statistics and achieve significant performance improvements on benchmark tasks.

e5-omni is a framework for omni-modal representation learning that provides explicit cross-modal alignment when mapping heterogeneous modalities—such as text, image, video, and audio—into a shared embedding space. Designed as a lightweight and modular wrapper around off-the-shelf vision–LLM (VLM) backbones, e5-omni addresses systematic limitations observed in existing contrastive learning pipelines for omni-modal embeddings by integrating modality-aware temperature calibration, hard negative curriculum learning with debiasing, and covariance-regularized batch whitening. Empirical results on benchmark tasks demonstrate that this methodology yields robust improvements over strong bi-modal and omni-modal baselines while maintaining architecture-invariance and efficient implementation (Chen et al., 7 Jan 2026).

1. Motivation and Systematic Challenges in Omni-Modal Embeddings

Omni-modal embedding models facilitate direct comparison of diverse modalities by encoding each input (e.g., queries, images, audio, video clips) into a unified vector space. Prevailing approaches—often derived from VLM bi-encoders fine-tuned with vanilla contrastive objectives—encounter several systemic shortcomings in multi-modal batch training:

1. Modality-dependent sharpness: Using a fixed softmax temperature $\tau$ for all similarity logits ignores that modality pairings differ in noise and alignment precision (e.g., text–text pairs are less noisy than text–audio). This results in inconsistent scale and learning dynamics depending on the modalities involved.
2. Imbalanced negative hardness: In mixed-modality mini-batches, most negatives are either trivially easy (e.g., uncorrelated audio and image) or, less commonly, highly confusing. Uniformly treating all negatives reduces the impact of the most informative training signals as trivial negatives quickly dominate.
3. Mismatched embedding statistics: Without explicit regularization, embeddings from different modalities exhibit mismatches in first- and second-order statistics (means and covariances), destabilizing cross-modal ranking and retrieval.

These challenges motivated the development of e5-omni, which provides plug-in remedies to align temperature scaling, balance negative sampling, and normalize geometry across modalities (Chen et al., 7 Jan 2026).

2. Core Methodological Components

The e5-omni approach introduces three plug-in modules that collectively regularize cross-modal learning dynamics without changing the underlying backbone architecture:

2.1 Modality-Aware Temperature Calibration

A trainable temperature vector $\boldsymbol{\tau} \in \mathbb{R}^{|\mathcal{M}_0|}$ is maintained for the candidate modality set $$\mathcal{M}_0 = \{\mathrm{T}, \mathrm{I}, \mathrm{A}, \mathrm{V}\}$$ (Text, Image, Audio, Video). For an input $x$ of modality composition $m(x) \subseteq \mathcal{M}_0$, define a normalized one-hot weight $w(x) \in \Delta^{|\mathcal{M}_0| - 1}$, then set

$$\tau(x) = \max(w(x)^\top \boldsymbol{\tau}, 10^{-6}),$$

and for any pair $(q, p)$, the pairwise temperature

$\tau(q, p) = \tfrac{1}{2}(\tau(q)+\tau(p)).$

The calibrated logit becomes

$$\ell(q, p) = \frac{\langle \mathbf{e}(q), \mathbf{e}(p) \rangle}{\tau(q, p)},$$

where $\mathbf{e}(\cdot) \in \mathbb{R}^D$ is the embedding output. This aligns loss gradients to the noise characteristics of each modality pair, improving learning stability (Chen et al., 7 Jan 2026).

2.2 Controllable Negative Curriculum with Debiasing

For batch size $B$ and $K$ mined hard negatives per query, the full similarity matrix is $S \in \mathbb{R}^{B \times (B+K)}$. At training step $t$,

$$\rho_t = \rho_{\mathrm{init}} + (\rho_{\mathrm{final}} - \rho_{\mathrm{init}}) \cdot \mathrm{clip}\!\left(\frac{t-t_0}{T-t_0}, 0, 1\right)$$

controls the curriculum, selecting only the hardest portion of negatives per row as training progresses. The Debiased Contrastive Learning (DCL) loss is

$$\mathcal{L}_{\mathrm{DCL}} = -\tfrac{1}{B} \sum_{i=1}^B \log \frac{\exp(S_{ii})}{\exp(S_{ii}) + \widetilde{N}_i},$$

where

$$\widetilde{N}_i = \max \left( \sum_{j \in \Omega_i \setminus \{i\}} \exp(S_{ij}) - \gamma_+\, \exp(S_{ii}), \,\epsilon \right),$$

$\gamma_+ \in (0,1)$ is the debiasing coefficient, and $\Omega_i$ indexes selected negatives. This mechanism emphasizes informative, hard negatives while immunizing against false negative bias (Chen et al., 7 Jan 2026).

2.3 Batch Whitening and Covariance Alignment

Stacking the batch matrices for queries $Q = [e(q_i)]_{i=1}^B$ and positives $P = [e(p_i^+)]_{i=1}^B$, e5-omni computes the empirical covariance $\Sigma$, applies the whitening transform $W = \Sigma^{-1/2}$, yielding $\widehat{Q} = W(Q)$, $\widehat{P} = W(P)$. The CORAL-style penalty,

$$\mathcal{L}_{\mathrm{coral}} = \frac{1}{4D^2} \lVert \mathrm{Cov}(\widehat{Q}) - \mathrm{Cov}(\widehat{P}) \rVert_F^2,$$

aligns the second-order statistics of different modalities to stabilize cross-modal retrieval (Chen et al., 7 Jan 2026).

The total loss is

$$\mathcal{L} = \mathcal{L}_{\mathrm{DCL}} + \lambda_{\mathrm{coral}} \mathcal{L}_{\mathrm{coral}}.$$

3. Implementation Details

e5-omni was developed atop the Qwen2.5-Omni backbone (7B parameters), using LoRA adapters for parameter-efficient fine-tuning. The embedding dimension is fixed at $D=768$. Training uses eight A100/H100 GPUs with a batch size of 20 per GPU and gradient accumulation of 2, for an effective batch of $B=320$. AdamW optimizer is used with an initial learning rate of $1\mathrm{e}{-4}$ and 0.5% linear warmup, for one epoch across a mixture of contrastive pairs.

Two mined hard negatives per query per dataset are included, alongside in-batch negatives. Initial temperatures are set $\tau_0 = 0.02$ and converge to learned $\tau = [0.0130, 0.0127, 0.0219, 0.0223]$ for $\{\mathrm{T}, \mathrm{I}, \mathrm{A}, \mathrm{V}\}$, respectively. The curriculum advances from $\rho_{\mathrm{init}} = 0.1$ to $\rho_{\mathrm{final}} = 0.5$ after $t_0 = 4000$ steps, with the DCL coefficient $\gamma_+=0.1$ and covariance weighting $\lambda_{\mathrm{coral}}=0.05$. Tuning utilizes 1K-sample validation splits per training set (Chen et al., 7 Jan 2026).

4. Experimental Evaluation

e5-omni was evaluated on the MMEB-V2 benchmark (78 tasks including image, video, visual-document retrieval, classification, and Q&A) and AudioCaps dataset (4.4K text–audio pairs, Recall@1).

MMEB-V2 Main Results

Model	Size	Image Hit@1	Video Hit@1	VisDoc NDCG@5	All Avg
e5-omni-vanilla-7B	7B	70.8	40.9	69.0	64.4
e5-omni-7B	7B	73.0	42.6	70.4	66.4
Best prior omni-modal	7B	71.3	47.5	67.1	64.5

A statistically significant improvement ($p<0.01$) is observed over all baselines.

AudioCaps Results

Model	Size	Recall@1
Tevatron-Omni	7B	34.0
LCO-EMB	7B	24.2
Omni-Embed-Nemotron	3B	20.5
e5-omni-7B	7B	37.7

The improvement over all baselines is significant at $p<0.05$.

Ablation Analysis

Component Removed	MMEB-V2 All Avg	AudioCaps R@1
e5-omni full	66.4	37.7
w/o modality-temp	65.7	36.6
w/o curriculum schedule	65.7	36.7
w/o DCL (w/ schedule only)	66.1	37.0
w/o whitening & CORAL	65.9	36.3

Each module contributes distinct performance gains.

5. Limitations and Future Prospects

e5-omni is focused on explicit similarity-geometry and optimization targets rather than broader aspects such as multi-step reasoning or compositional semantic understanding. Batch statistics (especially covariance estimation for whitening) can be noisy in small or imbalanced mini-batches, despite mitigation efforts. The model's current evaluation templates are centered on MMEB-V2 (text/image/video/VisDoc) and AudioCaps (text/audio); assessments on broader scenarios, such as long-range video/audio retrieval, multimodal QA, or web-scale use cases, remain future work.

Potential future directions include adaptive curriculum schedules (e.g., dynamically adjusting the fraction of hard negatives), per-modality decorrelation strategies extending beyond whitening, and expanding to additional modalities such as 3D point clouds or tactile sensor data (Chen et al., 7 Jan 2026).

6. Relationship to Multilingual and Uni-modal E5-family Models

e5-omni, though sharing nomenclature and certain high-level goals with the E5 and “mE5” model family (Wang et al., 2024), is architecturally and methodologically distinct. E5 and mE5 offer scalable bi-encoder text embedding architectures using recipes involving large-scale multilingual contrastive pre-training and supervised fine-tuning, with instruction-tuned variants for fine-grained user control. The E5-Omni family (mE5_small/base/large, mE5_large_instruct) is specialized for multilingual text rather than omni-modality and uses standard InfoNCE loss instead of the explicit cross-modal alignment modules introduced in e5-omni.

A plausible implication is that e5-omni’s cross-modal regularization recipe could be adapted to future multilingual multi-modal embedding scenarios, leveraging both explicit geometric alignment and multilingual supervision for even broader universality. However, this integration remains an open direction for the field.