Enhancing Multilingual Reasoning via Steerable Model Merging

Abstract: Model merging is an effective technique for composing the capabilities of a multilingual model and a reasoning model. It has achieved promising generalization in multilingual reasoning tasks by aligning feature spaces of different models. However, the merged single model often fails to address the conflicts between source models, leading to suboptimal performance. In other words, the one-size-fits-all merging strategy may not align with the characteristics of different inputs which may require prioritizing certain models over others. To this end, we propose a Steerable Model Merging (ST-Merge) framework to modulate the contribution of each source model. To realize this idea, we introduce a gated cross-attention mechanism to weight or filter the two attended source models in an adaptive manner. Extensive experiments demonstrate that ST-Merge consistently outperforms multiple strong baselines on four multilingual reasoning benchmarks across 21 different languages.

• The paper introduces ST-Merge, an adaptive, steerable merging framework that combines multilingual encoders with reasoning LLMs for enhanced cross-lingual generalization.
• It employs feature alignment and gated cross-attention to dynamically balance language grounding and reasoning, achieving consistent accuracy improvements across 21 languages.
• The approach offers minimal computational overhead and interpretable, input-dependent fusion weights, addressing the limitations of fixed merging strategies in diverse language settings.

Steerable Model Merging for Adaptive Multilingual Reasoning

Motivation and Problem Statement

The paper "Enhancing Multilingual Reasoning via Steerable Model Merging" (2606.19002) introduces a framework tackling the problem of reasoning generalization across languages using LLMs. Standard approaches—either direct LLM inference or fixed model merging with a multilingual encoder—fail to optimize the balance between reasoning and language grounding, especially for low-resource languages. The adaptive merging challenge arises from the intrinsic feature conflicts between specialized models: dominant LLMs exhibit strong reasoning capacity primarily in high-resource (e.g., English) contexts, while multilingual encoders better capture semantics for under-represented languages but can dilute reasoning fidelity.

Figure 1: ST-Merge modulates contributions from each source model, unlike fixed strategies, enabling input-specific fusion for multilingual reasoning.

The paper establishes that a static merging mechanism is fundamentally suboptimal. Manual experiments varying the fusion weights ($\omega_A$, $\omega_B$) reveal language-sensitive optima, suggesting the necessity for input-dependent model collaboration.

Steerable Model Merging Architecture

The proposed ST-Merge framework consists of two stages: feature space alignment and steerable gated cross-attention.

Feature Alignment: The multilingual encoder (mT5) yields language-invariant features, projected into the reasoning LLM space via a two-layer MLP. Only the representations are aligned; base weights remain frozen, promoting modularity and reproducibility.

Gated Cross-Attention: A cross-attention module, with reasoning features as queries and aligned multilingual features as keys/values, enables token-level contextual interaction. A learnable language embedding is concatenated with the global context, and an MLP with $1 + \tanh$ activation produces scalar weights for the two branches. The resultant fused representation steers the frozen LLM decoder, permitting adaptive bias toward either model based on the input.

Figure 2: Overview of the ST-Merge framework, highlighting cross-attention and input-aware gating for dynamic feature blending.

Empirical Results

ST-Merge is evaluated on four multilingual reasoning benchmarks: MGSM, MSVAMP, X-CSQA, and XNLI, covering 21 languages. Experimental protocols ensure architectural parity with baselines such as MindMerger [NEURIPS2024_3bf80b34], LayAlign [ruan-etal-2025-layalign], and MetaMath [DBLP:conf/iclr/YuJSYLZKLWL24].

Numerical Gains: ST-Merge consistently surpasses strong baselines, yielding average accuracy improvements of +1.7% (MGSM), +1.3% (MSVAMP), +1.5% (X-CSQA), and +1.5% (XNLI) over MindMerger. High-resource accuracy is maintained or improved (e.g., MGSM English: 68.0%). The steerable mechanism is robust both in low-resource and high-resource contexts.

Figure 3: Heatmap analysis shows language-specific optimal fusion weights, with ST-Merge converging to those optima in each case.

Ablation studies reinforce the architectural components' necessity: removal of language embeddings, cross-attention, or the gating network results in substantial accuracy degradation, confirming the functional importance of steerability and token-level contextual fusion.

Adaptive Weight Analysis and Generalization

The paper's gating visualization demonstrates ST-Merge's convergence toward language-sensitive optima—higher multilingual weights for underrepresented languages, and higher reasoning weights for English. The gating mechanism is interpretable and exhibits strong correlation with reasoning correctness.

Figure 4: t-SNE visualization validates ST-Merge's cross-lingual alignment: representations mix and overlap, compared to language-dependent clustering in MetaMath.

ST-Merge expands the LLM's multilingual generalization by adaptively re-weighting semantic versus reasoning centricity per input. Case studies show its capability to rectify incorrect answers produced by fixed merging, notably in low-resource language scenarios.

Practical and Theoretical Implications

Practically, ST-Merge demonstrates plug-and-play enhancement with negligible parameter and FLOP overhead (+0.17% parameters, +0.0016% FLOPs). The steerable design is agnostic to backbone architectures and may be incorporated into broader model merging paradigms for cross-lingual transfer, multimodal fusion, or domain generalization.

Theoretically, this work points to a paradigm where fine-grained, input-adaptive merging enables robust generalization without catastrophic forgetting or dominant source model bias. The gating mechanism provides a direct empirical knob for examining and controlling inductive bias across heterogeneous model components.

Future Directions

The current implementation operates at the embedding level and sequence-level granularity. Further granularity—token-level, step-aware guidance during decoding—could enhance fine-grained control and performance, especially in highly compositional tasks.

Additionally, the steerable merging principle may be extended to integrate additional modalities or architectures (e.g., multimodal LLMs, modular reasoning systems), offering a scalable path for unified reasoning across heterogeneous data spaces.

Conclusion

ST-Merge introduces a steerable, gated model merging mechanism that modulates the contributions of a multilingual encoder and reasoning LLM per input for multilingual reasoning tasks. This adaptive approach overcomes fixed-allocation conflicts, achieving consistent state-of-the-art results across multiple benchmarks and languages. Its design ensures scalable, generalizable enhancement of LLM multilingual capabilities with minimal computational overhead, and the steerable merging paradigm offers a promising avenue for future research in adaptive model composition and cross-lingual generalization.