Upcycling Large Language Models

Updated 1 February 2026

Upcycling Large Language Models is a set of techniques that repurpose pretrained dense models through modular expansion to conserve compute and boost domain specialization.
It involves methods such as expert grafting, dynamic routing, and embedding adaptation, which provide efficient performance gains without full retraining.
Empirical results show upcycling can deliver up to 20% improvements on benchmarks while reducing compute overhead, making deployment more scalable and cost-effective.

Upcycling LLMs refers to an array of techniques that repurpose, restructure, or augment already-trained dense LLMs—leveraging their sunk computational investment—to yield models with greater representational capacity, domain specialization, or efficiency, often through Mixture-of-Experts (MoE) mechanisms. Instead of full retraining, upcycling conserves prior model knowledge and focuses subsequent compute on modular expansion or specialization, resulting in substantial improvements in performance, scalability, memory footprint, and practical deployment flexibility. Methods range from architecture-level parameter grafting to embedding adaptation, efficient expert discovery, dynamic safety regulation, and domain/language extension.

1. Upcycling Principles and Motivation

Upcycling in LLMs is conceptually analogous to industrial upcycling, in which used materials are refashioned into products of higher value (Wang et al., 9 Oct 2025, He et al., 2024). For LLMs, upcycling repurposes a pretrained dense model’s parameters—particularly FFN/MLP weights—but can include attention weights, positional embeddings, or even entire layers (Zhang et al., 2024, Teo et al., 14 Mar 2025). Its core principle is function-preserving, efficient expansion: practitioners "graft" experts onto a model scaffold, initializing with existing weights and minimal additional random parameters (e.g., router networks), thus inheriting prior knowledge while allowing new specialization. This sharply reduces data and compute demands compared to training MoEs from scratch or continuing dense pretraining.

Upcycling is motivated primarily by the high cost and environmental impact of LLM pretraining (e.g., 10³–10⁴ PFLOPs for web-scale models (Komatsuzaki et al., 2022, Wang et al., 9 Oct 2025)) and by the opportunity to access capabilities or capacities unattainable under dense training constraints.

2. Architectures and Mechanisms for Upcycling

Most upcycling methods instantiate MoE architectures over selected submodules—typically FFN blocks, but also attention or entire transformer layers (Vavre et al., 2024, Zhang et al., 2024). A standard recipe is:

FFN Expert Duplication: Each dense FFN's weight matrices (W₁, W₂) are replicated E times per MoE layer, forming E parallel "experts" (He et al., 2024, Doubov et al., 2024). For attention upcycling, specialized multi-head parameters can be separately or jointly upcycled as “Mixture-of-Attention” experts (Zhang et al., 2024).
Router Integration: A learned router network maps incoming activations to expert selection probabilities via a softmax (or other dynamic gating) (Doubov et al., 2024, Vavre et al., 2024). At inference, only the top-K experts per token are activated ("sparse" compute), sharply reducing active FLOPs relative to total parameters.
Virtual Group and Weight Scaling: For fine-grained MoE (granularity G>1), virtual group initialization partitions FFN projections, preserves functional equivalence to dense layers, and requires cubic rescaling of weights for output magnitude consistency (He et al., 2024).
Expert Expansion and Parameter Diversity: Genetic algorithms and parameter merging ensure diverse expert initialization, maximizing specialization and utilization (Hui et al., 2024). Noise injection, layer copying, and cross-expert mixing are used for structural and functional diversity (Wang et al., 9 Oct 2025, Hui et al., 2024).

Specialized variants include layer-wise upcycling (MoLEx (Teo et al., 14 Mar 2025)), partial upcycling for modular safety (UpSafe°C (Sun et al., 2 Oct 2025)), and upcycling candidate tokens for query expansion (CTQE (Kim et al., 2 Sep 2025)).

3. Performance, Efficiency, and Empirical Scaling Laws

Upcycling typically achieves superior model performance to continued dense training and can require 15–50 % of the additional computational budget needed for dense quality increases (Doubov et al., 2024, Komatsuzaki et al., 2022, He et al., 2024). Key metrics and findings include:

Quality Gains

Sparse upcycling of 436 M and 1.4 B models delivers up to +20 % and +15 % relative improvements, respectively, on major downstream benchmarks versus continued pretraining (Doubov et al., 2024).
Upcycled Nemotron-4 15B models reach 67.6% MMLU, versus 65.3% for a densely continued counterpart (same token budget) (He et al., 2024).
MoE upcycles on T5 and ViT yield +1.7–2.0 points on SuperGLUE and ImageNet at ~40–60 % extra compute, outperforming scratch MoE for moderate budgets (Komatsuzaki et al., 2022).

Efficiency and Trade-offs

Per-token inference throughput drops 35–45 % due to sparse expert activation (i.e., slower inference for higher quality), with memory footprints rising 3–5× (Doubov et al., 2024, Vavre et al., 2024).
Empirical scaling laws show upcycled MoE loss curves $L(D_1, D_2)$ have an "interaction term" with sunk dense tokens $D_1$ limiting marginal returns on further upcycling—efficiency saturates at large budgets (Liew et al., 5 Feb 2025).
Upcycling is compute-optimal for moderate additional budgets but can be outperformed by from-scratch MoE training beyond a critical dataset size $D^*(N_1)$ (Liew et al., 5 Feb 2025).

Method	Quality Gain vs CPT	Inference Slowdown
Sparse Upcycling	+15–20%	–35–45%
Dense CPT	0%	0%
MoE-from-scratch	variable	similar

4. Specialized Upcycling: Scientific, Multilingual, Safety, and Token-Efficient Expansions

Scientific Domain Upcycling

Innovator applies a four-stage MoE upcycling paradigm to generalist LLMs: (1) induction of discipline experts, (2) fine-splitting experts along FFN axes, (3) science-aware router warm-up, (4) joint integration with general data. For Qwen2.5-7B, Innovator achieves a 25% improvement on 30 scientific benchmarks and preserves 99% performance in general tasks, activating only 13.3B of 53.3B total parameters per token (Liao et al., 24 Jul 2025).

Multilingual Expansion

MoE-LPR freezes all original parameters, injects new FFNs as experts, and performs post-pretraining only on expanded languages. A language-prior routing review (<1% replay data) steers original languages back to the original expert with ~96.6% retention, with minimal runtime overhead (Zhou et al., 2024).

Controllable Safety

UpSafe°C upcycles only safety-critical layers into sparse MoEs with added safety experts. A two-stage supervised training schedule and a safety temperature inference knob (τ∈[0,1]) provide dynamic Pareto-optimal control over safety-utility tradeoffs. Empirically, UpSafe°C achieves 100% safety rate on StrongReject, JailbreakBench, and matches baseline general accuracy (Sun et al., 2 Oct 2025).

Parameter- and Token-Efficient Query Expansion

CTQE upcycles candidate tokens discarded during LLM decoding as expansion terms for IR, attaining >2 point nDCG improvements on BM25 at <3% of the token cost of state-of-the-art multi-sample expansion baselines (Kim et al., 2 Sep 2025).

5. Hyperparameter Recipes and Best Practices

Key findings from systematic ablations and large-scale experiments include:

Initialization: For fine-grained MoE (E=8, G=8, T=8), use virtual group init for functional equivalence; set router weights uniformly inside groups. Merge weights using cubic scaling $\alpha = \sqrt[3]{E G^2/T}$ (He et al., 2024).
Learning Rate and Schedules: Reset LR to peak pretraining value for better escape from local minima; cosine annealing accelerates convergence (He et al., 2024).
Load Balancing and Routing: Auxiliary load balance losses with λ ∈ [1e–3, 1e–2] prevent expert collapse; softmax-then-topK routing empirically dominates topK-then-softmax (He et al., 2024).
Expert Count and Granularity: Upcycling with 8–64 fine-grained experts improves accuracy and convergence; diminishing returns above 128 due to memory and kernel bottlenecks.
Deployment: MoE-specific inference engines, memory partitioning, and activation count control (capacity factor CF) are critical in practice (Vavre et al., 2024).

6. Expansions, Layer Reduction, and Embedding-Level Upcycling

Layer-wise upcycling (MoLEx) constructs SMoEs by interpolating between original layers and mixtures of other layers in a model, with learned routing and negligible overhead; yields +0.6 points on GLUE and double-digit gains in zero-shot transfers (Teo et al., 14 Mar 2025). Structural slimming by layer cutting (e.g., retaining only 1–2 layers in GPT2-XL/OPT-1.3B) consistently retains or improves accuracy with 90%+ parameter savings for classification tasks (Yuan et al., 2024).

For cross-lingual upcycling, retraining only the lexical embedding matrix over a frozen transformer, followed by linear mapping for model scaling, achieves perplexities and human quality indistinguishable from scratch-trained models—drastically lowering adaptation cost for new languages (Vries et al., 2020).

7. Future Directions and Open Questions

Ongoing research probes:

Scaling Laws: More granular theoretical models linking compute, data, model capacity, and quality under various upcycling regimes (Liew et al., 5 Feb 2025).
Expert Specialization: Domain-specific expert induction, branching-train-mix, and adaptive expert expansion for continual learning (Hui et al., 2024, Wang et al., 9 Oct 2025).
Inference Optimization: Tailored inference kernels to mitigate throughput penalties, quantization, and sparse memory dispatch for large expert ensembles (Doubov et al., 2024).
Modular Control: Extension of upcycling safety control to bias mitigation, style transfer, and domain-specific guardrails (modular MoE upcycling) (Sun et al., 2 Oct 2025).
Cross-Task Utility: Evaluation of upcycling effects beyond classification—including generative, reasoning, retrieval, and multilingual understanding tasks.
Combination with PEFT: Orthogonality to PEFT (LoRA, adapters); joint application supports highly parameter-efficient fine-tuning (Teo et al., 14 Mar 2025).

Upcycling methods are rapidly evolving as the dominant paradigm for economical scaling, transfer, and specialization in LLMs. Practitioners are advised to benchmark architectures, compute allocations, and effectiveness using empirical scaling laws and robust ablations, with careful attention to deployment constraints and downstream requirements.