Grow, Don't Overwrite: Fine-tuning Without Forgetting

Abstract: Adapting pre-trained models to specialized tasks often leads to catastrophic forgetting, where new knowledge overwrites foundational capabilities. Existing methods either compromise performance on the new task or struggle to balance training stability with efficient reuse of pre-trained knowledge. We introduce a novel function-preserving expansion method that resolves this dilemma. Our technique expands model capacity by replicating pre-trained parameters within transformer submodules and applying a scaling correction that guarantees the expanded model is mathematically identical to the original at initialization, enabling stable training while exploiting existing knowledge. Empirically, our method eliminates the trade-off between plasticity and stability, matching the performance of full fine-tuning on downstream tasks without any degradation of the model's original capabilities. Furthermore, we demonstrate the modularity of our approach, showing that by selectively expanding a small subset of layers we can achieve the same performance as full fine-tuning at a fraction of the computational cost.

• The paper presents a network expansion method on MLP layers that preserves original model functions while entirely avoiding catastrophic forgetting.
• It introduces two fine-tuning protocols, G-Freeze and G-Train, which selectively update new parameters to maintain baseline performance and adapt to new tasks.
• Empirical results demonstrate that targeted expansion of select layers achieves performance parity with full fine-tuning, while completely retaining foundational knowledge.

Function-Preserving Network Growth for Catastrophic Forgetting: An Expert Analysis

Introduction

The paper "Grow, Don't Overwrite: Fine-tuning Without Forgetting" (2603.08647) presents a principled and modular methodology to completely circumvent catastrophic forgetting during downstream adaptation of large pretrained Transformer-based models. The central innovation is a function-preserving network expansion strategy operating on MLP submodules, which enables performance parity with full fine-tuning on novel tasks while identically preserving the foundational capabilities of the original model. The approach is computationally and memory efficient, allows for expansion of only targeted layers, and provides empirical evidence for scaling laws that link problem complexity to required parameter growth. The authors deliver strong empirical and representational evidence to support the total elimination of the forgetting-performance trade-off, advancing both theoretical understanding and practical methodology.

Methodology: Function-Preserving MLP Expansion

Standard fine-tuning of deep networks induces representational drift and parameter overwrites that erase pre-trained knowledge, a challenge only partially mitigated by regularization or replay techniques. The method introduced here leverages insight from functional compositionality and network morphisms to add capacity in a mathematically-consistent manner, specifically targeting the up- and down-projection matrices of MLPs within each Transformer block.

The expansion procedure operates as follows: given an up-projection weight $W_n^{(1)}$, the hidden dimension $p$ is doubled by horizontally stacking two copies of $W_n^{(1)}$. The associated down-projection matrix $W_n^{(2)}$ is vertically concatenated with itself, with each block scaled by $1/2$, precisely compensating for the increased activation width. This procedure is proven to be function-preserving at initialization, ensuring that, prior to further training, the expanded model outputs are exactly those of the original.

Figure 1: (a) Schematic of the expansion by duplication and scaling for function preservation. (b) Fine-tuning strategies: G-Freeze (training only new parameters) and G-Train (training the up-projection while freezing the down-projection).

Two fine-tuning protocols are considered post-expansion:

• G-Freeze: Only parameters introduced by the expansion are trained, guaranteeing maximal representation stability.
• G-Train: For tasks requiring higher plasticity (notably those with intermediate domain shift but elevated reasoning demands), the full up-projection matrix is trained with the down-projection and all non-expanded parameters frozen.

The strategy supports arbitrary expansion factors ($k \geq 2$), but $k=2$ realizes the optimal empirical trade-off.

Empirical Evaluation and Results

Knowledge Retention and Task Transfer

Benchmarked against domain-shifted tasks (e.g., FR-EN translation/MTNT, science entailment/SciTail, science QA/QASC, and MathQA), the approach produces compelling results: for all tasks, downstream performance is indistinguishable or superior to standard fine-tuning, while proxy measures of pretraining knowledge (e.g., WinoGrande accuracy) exhibit zero degradation. Notably, simple SFT produces near-total collapse on the original domain when the fine-tuning target is disjoint. The growing method's retention is invariant to scale, as validated on both 1B and 4B parameter Gemma architectures.

Figure 2: Downstream task performance and base capability retention: growing matches SFT on new tasks while fully preserving pretraining skills; SFT induces severe forgetting.

Modularity and Layer Selection

The method enables expansion of a selected subset of layers, identified either heuristically (by update magnitude) or by more sophisticated localization of task-relevant modules. Empirical ablation demonstrates that expanding as few as 9–10 layers (∼30% of parameters) suffices to recover full fine-tuning performance, reducing computational requirements without loss of efficacy.

Figure 3: Targeted growth of 10 layers recapitulates the performance of full-layer growth, confirming parameter efficiency and modularity.

Scaling Laws and Task Complexity

Ablations varying the number of expanded layers $N$ reveal that performance on complex tasks, especially mathematical reasoning, exhibits clear positive scaling with increasing capacity. Simple tasks (e.g., entailment) saturate rapidly, whereas higher-order reasoning demands distributed capacity increases across most or all layers.

Figure 4: Task performance as a function of number of expanded layers. Scaling is most pronounced on more complex tasks.

Analysis of the effective rank of up-projection weight updates demonstrates that task complexity dictates the required breadth of network adaptation: complex tasks (MathQA) require high-rank updates across essentially all layers, supporting the observed scaling phenomena.

Figure 5: Layerwise effective-rank of weight updates—complex reasoning tasks entail globally distributed, high-rank modifications.

Representation Stability

Comparative analysis of function vectors (FV) for the original and fine-tuned models demonstrates that the proposed method preserves internal representational subspaces, as measured by FV cosine similarity (0.95 for entailment, vs. 0.28 for SFT) and overlap in causal attention heads.

Additional Ablations

• Zero-initialization of new parameters results in either poor adaptation (if all parameters are trained) or rapid forgetting (if only new parameters are trained).
• Expansion of attention modules (e.g., increasing the number of heads or their dimension) yields suboptimal results relative to MLP expansion.
• Findings are robust at increased scale, verified on larger (4B) Gemma models.

Theoretical and Practical Implications

The method eliminates the plasticity-stability dilemma by directly decoupling new skill acquisition from prior knowledge retention in a single network instance, in contrast to approaches that rely on regularization, parameter isolation, replay, or multi-model ensembles. This provides a tractable and interpretable principle for continual/lifelong learning in transformer architectures, and connects with recent advances in functional modularity, circuit analysis, and model editing.

Practically, this technique improves model utility for sequential multitask adaptation, domain transfer, and scientific specialization, preventing catastrophic loss of previously valuable competencies. The modular growth mechanism naturally enables skill localization and potential future "skill merging" approaches, wherein the expert subnetworks for multiple downstream domains could be composed without retraining.

Speculation on Future Directions

Future developments could generalize this method to other architectural submodules, possibly integrating with PEFT/LoRA methods for further efficiency. Incorporating more advanced strategies for identifying optimal layers for growth (e.g., via gradient-based attribution, functional scan techniques) could further reduce costs and optimize distribution of computation per task. Finally, the framework provides a rigorous foundation for interpretable, transparent continual learning systems, and could inform work in mission-critical, safety-sensitive, or regulated domains where non-forgetting is a hard requirement.

Conclusion

The function-preserving MLP expansion paradigm advanced in this work offers a systematic solution to the catastrophic forgetting phenomenon in large-scale transformers. The approach is backed by comprehensive empirical and representational analysis, demonstrating full retention of foundational skills alongside optimal downstream task performance, all within a single, modular, efficiently parameterized model. Theoretical and experimental results support the broader principle that targeted capacity augmentation—when carefully engineered—can fundamentally reshape the limitations of neural network continual learning.