jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval

Abstract: We introduce jina-embeddings-v4, a 3.8 billion parameter multimodal embedding model that unifies text and image representations through a novel architecture supporting both single-vector and multi-vector embeddings in the late interaction style. The model incorporates task-specific Low-Rank Adaptation (LoRA) adapters to optimize performance across diverse retrieval scenarios, including query-document retrieval, semantic text similarity, and code search. Comprehensive evaluations demonstrate that jina-embeddings-v4 achieves state-of-the-art performance on both single-modal and cross-modal retrieval tasks, with particular strength in processing visually rich content such as tables, charts, diagrams, and mixed-media formats. To facilitate evaluation of this capability, we also introduce Jina-VDR, a novel benchmark specifically designed for visually rich image retrieval.

• The paper introduces a 3.8B parameter model that unifies text and image embeddings with both single-vector and multi-vector outputs for versatile retrieval tasks.
• The paper employs dual output modes enhanced by LoRA adapters and a two-stage training paradigm that leverages contrastive and task-specific losses to achieve state-of-the-art benchmark results.
• The paper's unified transformer architecture effectively minimizes the modality gap, delivering superior cross-modal alignment and multilingual performance for complex document retrieval.

Universal Multimodal Multilingual Embeddings: An Analysis of jina-embeddings-v4

The paper "jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval" (2506.18902) presents a 3.8B parameter embedding model that unifies text and image representations, supporting both single-vector and multi-vector (late interaction) embeddings. The model is designed for a broad spectrum of retrieval tasks, including cross-modal semantic similarity, information retrieval, and code search, and introduces the Jina-VDR benchmark for visually rich document retrieval.

Model Architecture and Training Paradigm

jina-embeddings-v4 is built on the Qwen2.5-VL backbone, extending it with:

• Dual Output Modes: Users can select between single-vector (2048d, truncatable) and multi-vector (128d per token) outputs. The single-vector mode is optimized via Matryoshka Representation Learning, enabling truncation with minimal performance loss.
• LoRA Adapters: Three task-specific Low-Rank Adaptation (LoRA) adapters are provided for asymmetric retrieval, symmetric semantic similarity, and code retrieval. These adapters are lightweight (60M parameters each) and can be selected at inference time.
• Unified Multimodal Processing: Both text and images are processed through a shared transformer path, with images converted to "image tokens" and passed to the LLM, minimizing the modality gap.

The training procedure involves two phases: initial contrastive pair training (text-text and text-image) with InfoNCE and Matryoshka loss, followed by task-specific fine-tuning of the LoRA adapters using hard negatives and specialized loss functions (e.g., CoSENT for similarity).

Jina-VDR Benchmark

The Jina-VDR benchmark is introduced to evaluate retrieval on visually rich documents, extending prior work (ViDoRe) with:

• 30 new tasks, including real-world and synthetic data.
• Multilingual and multi-domain coverage (e.g., legal, historical, marketing, technical).
• Diverse document types (charts, tables, maps, rendered markdown, scanned documents).
• Non-question queries and LLM-generated queries for broader coverage.

This benchmark enables comprehensive evaluation of models' ability to align and retrieve across complex, mixed-modality documents.

Empirical Results

Retrieval and Similarity

jina-embeddings-v4 demonstrates strong performance across a wide range of benchmarks:

Model	J-VDR	ViDoRe	CLIPB	MMTEB	MTEB-en	COIR	LEMB	STS-m	STS-en
jina-embeddings-v4 (dense)	72.19	84.11	84.11	66.49	55.97	71.59	67.11	72.70	85.89
jina-embeddings-v4 (late)	79.29	90.17	—	—	—	—	—	—	—
jina-embeddings-v3	47.06	26.02	—	58.58	54.33	55.07	55.66	75.77	85.82
gemini-embedding-001	—	—	—	67.71	64.35	73.11	—	78.35	85.29

• Jina-VDR and ViDoRe: jina-embeddings-v4 achieves the highest scores, especially in late interaction mode (nDCG@5: 79.29 on Jina-VDR, 90.17 on ViDoRe), outperforming both CLIP-style and other VLM-based models.
• Text Retrieval (MTEB/MMTEB): Performance is competitive with or superior to other state-of-the-art models, particularly for long-document retrieval (LEMB) and English STS tasks.
• Code Retrieval (CoIR): The code adapter achieves strong results, though specialized models (e.g., voyage-code) retain an edge on code-specific tasks.

Cross-Modal Alignment and Modality Gap

The model's unified encoder architecture sharply reduces the modality gap, as evidenced by:

• Cross-Modal Alignment Scores: On Flickr30K and MSCOCO, jina-embeddings-v4 achieves alignment scores of 0.71 and 0.72, compared to 0.15 and 0.14 for OpenAI CLIP.
• Distribution Analysis: The cosine similarity distributions for positive and negative pairs are more distinct and better separated than in CLIP-style models, indicating more effective use of the embedding space and improved cross-modal retrieval precision.

Multilingual and Multimodal Generalization

The model demonstrates robust performance across languages and modalities, with multilingual retrieval and semantic similarity scores that are consistently high across diverse datasets (see detailed tables in the appendix). The inclusion of LoRA adapters allows for efficient task specialization without significant memory overhead.

Implementation and Deployment Considerations

• Model Size: At 3.8B parameters plus adapters, the model is suitable for deployment on modern GPUs (A100/H100 class) and can be quantized for more resource-constrained environments.
• Adapter Selection: The LoRA adapters can be dynamically selected at inference, enabling a single deployment to serve multiple retrieval scenarios (text, image, code).
• Truncatable Embeddings: Matryoshka learning allows practitioners to trade off embedding size and retrieval accuracy, optimizing for storage or bandwidth constraints.
• Late Interaction: Multi-vector output enables ColBERT-style late interaction retrieval, which, while more computationally intensive, yields higher precision for complex queries and visually rich documents.

Theoretical and Practical Implications

• Unified Embedding Space: The reduction of the modality gap and improved cross-modal alignment suggest that unified transformer-based architectures are preferable to dual-encoder designs for general-purpose multimodal retrieval.
• Task Modularity: The LoRA-based adapter approach demonstrates that large, generalist models can be efficiently specialized for diverse retrieval tasks without retraining the full model.
• Benchmarking Visually Rich Documents: The introduction of Jina-VDR addresses a critical gap in evaluation, enabling more realistic assessment of models' capabilities in enterprise and real-world document search scenarios.

Future Directions

• Model Compression: Further work on quantization and distillation could yield smaller, more efficient variants suitable for edge deployment.
• Expanded Multilinguality: Extending language coverage, especially for low-resource languages, remains an open challenge.
• Unified Instruction Tuning: Integrating instruction-based retrieval (as in recent instruction-tuned embedders) could further improve generalization and user control.
• Broader Modalities: Extending the unified embedding approach to audio, video, and other modalities would further enhance the universality of the model.

Conclusion

jina-embeddings-v4 represents a significant advance in universal embedding models, offering a practical, high-performance solution for multilingual, multimodal, and multi-task retrieval. Its architecture and training paradigm provide a blueprint for future developments in unified representation learning, with strong empirical results and clear applicability to real-world information retrieval challenges.