Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs

Abstract: Composing independently trained LoRA adapters into a single LLM is useful for multi-domain adaptation, especially when the original training data cannot be shared. A common approach is to use MoE-style routing over LoRA experts, but for frozen pretrained adapters, soft weighted combinations can change the unit-scale additive update under which each LoRA module was originally trained. We propose \textbf{Hard-Routed MoR-LoRA}, a two-stage framework for composing frozen reasoning LoRA experts through unit-scale hard selection. First, domain-specific LoRA adapters are trained independently using reinforcement learning from verifiable feedback to obtain reasoning experts. Then, all experts are frozen, reasoning traces are distilled from them, and only a lightweight shared router together with a small attention LoRA is trained for integration. The router selects exactly one expert per token using hard top-1 routing, while a straight-through estimator enables gradient-based training. Experiments across five benchmarks, multiple model scales, and additional model families show that Hard-Routed MoR-LoRA preserves expert behavior while requiring substantially fewer trainable parameters than soft-routing mixture baselines. Our analysis further shows that normalized soft mixtures often concentrate most routing mass on a single expert, suggesting that hard unit-scale routing provides a simple and efficient abstraction for frozen LoRA expert composition.

• The paper introduces Hard-Routed MoR-LoRA, a method for integrating frozen, reasoning-specialized LoRA adapters using hard token-level routing to preserve expert behavior.
• It employs decoupled stages of domain-specialized training via RLVF and frozen expert integration, yielding notable accuracy improvements on benchmarks such as GSM8K and ARC-Challenge.
• The approach achieves high parameter efficiency by updating only a lightweight router and a minimal attention-layer LoRA, matching or exceeding soft-routing baselines.

Hard-Routed Mixtures of Reasoning LoRAs: An Expert-Level Synopsis

Motivation and Problem Definition

Independently trained Low-Rank Adaptation (LoRA) adapters enable modular, parameter-efficient domain adaptation for LLMs, especially when direct data sharing is infeasible due to privacy or regulatory issues. A natural integration strategy involves combining these frozen, pretrained adapters into a unified model capable of handling diverse domains. Existing techniques often rely on Mixture-of-Experts (MoE) routing with soft expert combination. However, this introduces a fundamental scale-mismatch: LoRA modules are trained with unit-scale updates, while MoE mixtures weigh expert outputs according to routing probabilities—distorting the learned expert behavior and necessitating retraining with additional parameters, as seen in approaches like LoRA-Mixer.

This work addresses the challenge of integrating frozen, reasoning-enhanced LoRA experts under strict modularity and parameter efficiency constraints. The solution, Hard-Routed MoR-LoRA, proposes explicit hard expert selection via a lightweight router and preserves the behavioral integrity of each adapter.

Methodological Framework

The pipeline comprises two decoupled stages:

1. Domain-specialized LoRA training via RLVF: Each adapter is independently trained using Reinforcement Learning from Verifiable Feedback (RLVF), leveraging automatically verifiable rewards (e.g., answer correctness). This facilitates specialization in domain-specific reasoning and maintains chain-of-thought (CoT) behaviors in instruction-tuned backbones.
2. Frozen expert integration with hard routing: A shared token-level router (linear, layer-shared) is trained—using distilled expert traces and next-token likelihood—to select exactly one expert per token. A straight-through estimator (STE) provides differentiable routing by combining discrete expert selection in the forward pass with surrogate gradients.

No expert retraining occurs during integration, and only the router and a minimal attention-layer LoRA are updated, achieving parameter efficiency independent of the number of experts.

Empirical Results: Reasoning and Integration

Quality of Reasoning Experts

Training with RLVF yields substantial performance gains on reasoning-intensive benchmarks (e.g., GSM8K, ARC-Challenge) for models with sufficient reasoning capacity (3B and 8B). In particular, the 3B model improves average accuracy from 68.60 (SFT) to 72.69 (RLVF); the 8B model from 72.72 to 79.72. This improvement is attributed to RLVF incentivizing verifiable correctness rather than imitation, thereby reinforcing multi-step and CoT reasoning. For low-capacity models (1B), RLVF provides limited benefits due to insufficient expressivity.

Figure 1: RLVF improves performance for 3B and 8B models but not for 1B, suggesting that reasoning-oriented training benefits models with sufficient capacity while preserving CoT behavior.

Hard Routing vs. Soft Routing in Expert Composition

Hard-Routed MoR-LoRA achieves comparable or superior integration performance to soft-routing baselines (e.g., LoRAMixer), but with orders of magnitude fewer trainable parameters. For example, in integration using 3B RLVF-trained experts, Hard-Routed MoR-LoRA attains 72.07 average accuracy with only ~73M trainable parameters, whereas LoRAMixer Top-1 achieves 65.67 with ~606M parameters. The largest gains are observed in domains benefiting from specialized reasoning.

Figure 2: Across model scales and expert types, Hard-Routed MoR-LoRA achieves comparable or better performance than soft routing despite not retraining experts, indicating that routing quality dominates expert adaptation.

Prompt-level selection—where a single expert is chosen for the entire prompt—can be highly competitive in single-domain settings. However, only token-level hard routing supports mixed-domain inputs, enabling question-wise expert routing within a prompt.

Analysis of Soft Routing Collapse

Empirical routing distributions reveal that normalized soft mixtures, even with top-2 selection, frequently collapse to nearly single-expert selection. On GSM8K, the average dominant routing weight is 0.71, and average routing entropy is low (0.6245). Thus, the bulk of routing capacity is concentrated on one expert, essentially mimicking hard unit-scale selection but with unnecessary computational complexity.

Figure 3: The dominant routing weight under top-2 normalized routing, showing that soft top-2 routing often behaves like near single-expert selection with an average dominant weight of 0.71.

Parameter Efficiency and Integration Budget

The parameter efficiency stems from updating only the router and a minimal attention LoRA. Experiments demonstrate that only a few integration samples per domain (≤1000) suffice to attain nearly maximal integration performance, since supervision is needed only for learning expert selection, not reasoning content.

Further ablations demonstrate (1) negligible Stage II gains from RLVF over standard distillation; (2) prompt-level routing matches token-level only in strictly single-domain contexts; (3) hard unit-scale routing must be enforced during both training and inference for effective transfer.

Implications and Future Directions

This analysis underscores a crucial separation between reasoning acquisition and integration in modular parameter-efficient adaptation. The ability to robustly compose frozen reasoning LoRA experts without altering their parameters offers significant scalability and privacy/compliance benefits for model deployment in federated or regulated environments.

Theoretical implications include the effectiveness of simple, discrete routing in capturing the practical behavior of probabilistic mixtures and suggesting a modular future for LLM adaptation—where selection, not retraining, is paramount. Practically, the method can enable efficient, traceable, and scalable expert reuse when original training data is inaccessible.

Potential research directions include generalized routing architectures for large, diverse expert pools; federated adapters with privacy-preserving integration; adaptive abstention or fallback routes through the unadapted backbone; and explicit mixed-domain training for dynamic expert switching.

Conclusion

Hard-Routed MoR-LoRA provides a robust framework for integrating independently trained, reasoning-specialized LoRA adapters under strict modularity. By reframing the integration problem as expert selection rather than adaptation, it preserves expert behavior, achieves strong parameter efficiency, and matches or exceeds the performance of traditional soft-routing and retraining-intensive methods. Empirical evidence demonstrates strong, scalable multi-domain reasoning and efficient transfer—providing a practical basis for future modular LLM systems and adaptive AI deployment.