OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging

Abstract: Foundation models update slowly due to resource-intensive training, whereas domain-specific models evolve rapidly between releases. Model merging seeks to combine multiple expert models into a single, more capable model, reducing storage and serving costs while supporting decentralized development. Despite its potential, previous studies have primarily focused on merging visual classification models or LLMs for code and math tasks. Recently, Multimodal LLMs (MLLMs) that extend LLMs through large-scale multimodal training have gained traction. However, there lacks a benchmark for model merging research that clearly divides the tasks for MLLM training and evaluation. In this paper, $\textbf{(i)}$ we introduce a model merging benchmark for MLLMs, which includes multiple tasks such as VQA, Geometry, Chart, OCR, and Grounding, studying both LoRA and full fine-tuning models. Moreover, we explore how model merging can combine different modalities (e.g., vision-language, audio-language, and video-LLMs), moving toward the Omni-LLM. $\textbf{(ii)}$ We implement 10 model merging algorithms on the benchmark. Furthermore, we propose a novel method that removes noise from task vectors and robustly optimizes the merged vector based on a loss defined over task vector interactions, achieving an average performance gain of 2.48%. $\textbf{(iii)}$ We find that model merging offers a promising way for building improved MLLMs without requiring training data. Our results also demonstrate that the complementarity among multiple modalities outperforms individual modalities.

• The paper presents OptMerge, a data-free model merging framework that employs SVD-based truncation and optimized initialization to integrate multimodal LLMs without retraining.
• It achieves a 4.65% performance boost over state-of-the-art methods on tasks like VQA, geometry, and chart analysis while lowering computational costs.
• Empirical findings underscore the importance of selecting experts with minimal parameter drift to ensure stable, noise-resistant task vector integration across modalities.

OptMerge: Unifying Multimodal LLM Capabilities via Model Merging

Introduction and Motivation

The paper presents OptMerge, a comprehensive framework for data-free model merging in the context of Multimodal LLMs (MLLMs). The motivation stems from the stagnant update cycles of general-purpose foundation models contrasted with rapid advancements in domain-specific expert models due to resource-intensive training bottlenecks. Model merging is posited as a scalable solution to combine complementary, fine-tuned expert models, thus unifying multiple capabilities or modalities in a single deployment. This is achieved without retraining or access to original data, thereby reducing both computational burden and the cost of serving diverse applications.

Model Merging Algorithms and Task Vector Theory

The authors provide a systematic categorization of static, data-free model merging algorithms. Four principal types are enumerated:

1. Linear Interpolation Methods: Direct averaging or arithmetic combination of task vectors, which are parameter differences between fine-tuned models and their shared base.
2. Sparsification-based Methods: Reduce redundancy or interference in task vectors via sparsity constraints and selective parameter inclusion.
3. SVD-based Methods: Exploit low-rank structure, orthogonalize task vectors, and isolate dominant singular components to mitigate task interference.
4. Optimization-based Methods: Frame the merging procedure as a parameter-space optimization problem, typically minimizing layer-wise interference metrics over task vectors using gradient descent.

A novel contribution here is a theoretical upper bound relating merging loss to the learning rate and number of fine-tuning iterations ($\mathcal{O}(\eta T)$), formalizing the notion that excessive fine-tuning increases parameter drift and complicates task vector integration.

Figure 1: Task vector magnitude distribution in InternVL2.5 reveals right-skewed fine-tuning changes, vital for successful model merging.

Further, empirical analysis using Frobenius norms per layer corroborates that effective merging requires closely clustered expert models in parameter space. For low-rank adaptation approaches like LoRA, task vectors cluster in restricted subspaces, amplifying the need for rank-aware merging strategies.

OptMerge Methodology: Robust Optimization of Task Vectors

The key contribution is OptMerge, instantiated to address discovered instability and noise amplification in existing data-free model merging workflows. Two specialized instantiations are described:

• Full Fine-Tuned Models: OptMerge executes SVD on centered task vectors, truncating noise-dominated singular components. The optimization loss leverages only dominant components, analogous to PCA, thereby preserving shared knowledge and reducing destructive interference.
• LoRA Fine-Tuned Models: OptMerge circumvents null-space gradient sparsity by initializing with averaged vectors and switching to SGD (exploiting implicit regularization). Direct SVD-based truncation maintains stable vector norms while avoiding shortcuts that inflate parameter magnitudes.

Figure 2: When optimizing the interference loss, merged task vectors can take degenerate shortcuts by increasing their norm—OptMerge regularizes this optimization.

Ablation analysis shows that initializing merge vectors and applying low-rank truncation offer a 4.65% average performance boost over previous state-of-the-art optimization-based merging algorithms.

Benchmark Construction and Evaluation Protocol

The authors introduce a meticulously curated model merging benchmark addressing five canonical MLLM capabilities: VQA, Geometry, Chart, OCR, and Grounding, with each backed by over 100k samples for supervised fine-tuning. Two base models—InternVL2.5 for instruction-following and Qwen2-VL-7B for domain-general tasks—are selected with both LoRA and full fine-tuning variants.

Downstream evaluation benchmarks are chosen to target specific subtasks rather than general holistic capability assessments, enabling nuanced analysis post-merging. Modality merging further extends the approach to include vision, audio, and video-language encoders, demonstrating the capacity for integrating tri-modal information streams.

Empirical Findings and Strong Numerical Claims

Experimental results yield several substantive findings:

• Model merging can outperform expert models and mixture multi-task training. For example, OptMerge merged Qwen2-VL achieves geometry scores of 51.05 and 40.79, compared to individual experts' 42.50 and 28.95, and chart scores of 79.76 against a 61.08 baseline.
• Optimal merging does not coincide with maximum fine-tuning steps—excessive fine-tuning leads to degraded merging performance due to parameter drift, as visualized by convergence plots and method comparisons.

  Figure 3: CLIP accuracy across eight datasets converges with training; fine-tuning does not guarantee better merging.

  

Figure 4: Across eight datasets, merging accuracy rises then falls with more fine-tuning—contradicting standard assumptions about model quality.

• Modality merging achieves higher average accuracy (up to 67.00%) than naively composing or averaging activations across independently trained vision/audio/video models.
• OptMerge maintains parameter norm stability, preventing collapse in merged LLMs and successfully incorporating knowledge from disparate domains and modalities.

Computational cost analysis reveals orders-of-magnitude savings: model merging completes in ≤4 hours and ≤22GB memory for 7B models, compared to ~25 hours and >240GB for multi-task mixture training.

Practical and Theoretical Implications

OptMerge demonstrates that data-free merging of expert MLLMs can outstrip multi-task mixture training in both effectiveness and computational tractability. This offers a flexible paradigm for continual capability integration and decentralized development, as open-source communities routinely contribute domain-specialized checkpoints. The benchmark provides a robust platform for fair, replicable evaluation of future model merging algorithms.

Theoretical results clarify the constraints introduced by parameter drift, informing best practices: select expert models with minimal fine-tuning divergence from base parameters, and prioritize less aggressively fine-tuned models for merging.

On a broader scope, this encourages the pursuit of omni-modal LLMs assembled without re-training, where capabilities are modular, extensible, and privacy-preserving. Open questions remain regarding generalization on multilingual data, further scaling, and reasoning-focused MLLM merging.

Conclusion

OptMerge unifies MLLM capabilities and modalities through principled, data-free model merging, introducing a stable optimization approach that denoises task vectors, regularizes parameter norms, and consistently achieves superior downstream performance. The main takeaways include the efficacy of low-rank approximation, initialization strategies in optimization, and the importance of judicious expert model selection. Future directions involve scaling to larger checkpoints, deeper investigations into reasoning and multilingual integration, and enhanced benchmarks for evaluating omni-model alignment.