Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models

Abstract: Despite the success, the process of fine-tuning large-scale PLMs brings prohibitive adaptation costs. In fact, fine-tuning all the parameters of a colossal model and retaining separate instances for different tasks are practically infeasible. This necessitates a new branch of research focusing on the parameter-efficient adaptation of PLMs, dubbed as delta tuning in this paper. In contrast with the standard fine-tuning, delta tuning only fine-tunes a small portion of the model parameters while keeping the rest untouched, largely reducing both the computation and storage costs. Recent studies have demonstrated that a series of delta tuning methods with distinct tuned parameter selection could achieve performance on a par with full-parameter fine-tuning, suggesting a new promising way of stimulating large-scale PLMs. In this paper, we first formally describe the problem of delta tuning and then comprehensively review recent delta tuning approaches. We also propose a unified categorization criterion that divide existing delta tuning methods into three groups: addition-based, specification-based, and reparameterization-based methods. Though initially proposed as an efficient method to steer large models, we believe that some of the fascinating evidence discovered along with delta tuning could help further reveal the mechanisms of PLMs and even deep neural networks. To this end, we discuss the theoretical principles underlying the effectiveness of delta tuning and propose frameworks to interpret delta tuning from the perspective of optimization and optimal control, respectively. Furthermore, we provide a holistic empirical study of representative methods, where results on over 100 NLP tasks demonstrate a comprehensive performance comparison of different approaches. The experimental results also cover the analysis of combinatorial, scaling and transferable properties of delta tuning.

• The paper introduces delta tuning, a parameter-efficient strategy that adapts only a small subset of model parameters for diverse NLP tasks.
• It presents a unified taxonomy grouping addition-based, specification-based, and reparameterization-based methods, supported by optimization and control theory frameworks.
• Empirical studies on over 100 NLP tasks reveal that delta tuning yields competitive accuracy with reduced computational resource usage and improved transferability.

Delta Tuning: An Analytical Review of Parameter-Efficient Methods for Pre-trained LLMs

Introduction and Motivation

The paper provides a detailed investigation into the scaling constraints of full-model fine-tuning for large pre-trained LLMs (PLMs), identifying adaptation costs in computation and storage as primary bottlenecks for both academic and industrial NLP research. Through a survey of recent conference publications, the authors reveal that only a small fraction (0.5%--4%) of works employ large-scale PLMs (>1B parameters) in experiments, highlighting the practical challenges of fully parameterized model adaptation. To address these challenges, the paper introduces and formalizes the notion of "delta tuning," a paradigm that restricts adaptation to a small subset of model parameters, thus enabling parameter-efficient model specialization for diverse downstream tasks.

Types of Delta Tuning Methods

The authors propose a unified taxonomy for delta tuning methods:

• Addition-based: Inject extra trainable components, such as adapters or prompts, without modifying the core model. Examples include adapter-based tuning and prompt/prefix tuning.
• Specification-based: Select and tune only a subset of inherent parameters (e.g., biases or final layers), freezing the rest. BitFit is a canonical instance.
• Reparameterization-based: Replace original parameters or their updates with low-rank decompositions or subspace projections, exploiting intrinsic task-adaptation low dimensionality. Notable examples include LoRA and intrinsic dimension-based methods.

Each class is grounded in its operational constraints—whether tunable parameters are newly introduced, selected, or reparameterized for adaptation efficiency.

Theoretical Frameworks for Delta Tuning

Two complementary theoretical paradigms are developed:

• Optimization Perspective: Delta tuning is analyzed as subspace optimization in high-dimensional parameter spaces, justified by empirical findings that adaptation often occurs on low-dimensional manifolds. The authors derive bounds on performance discrepancy versus full fine-tuning, with error terms governed by quality of low-rank approximations and sensitivity to model initialization.
• Optimal Control Perspective: The adaptation problem is reinterpreted through discrete-time optimal control, where delta-parameters serve as task-specific controllers for the PLM. Through Pontryagin's Maximum Principle and method of successive approximations, the forward and backward propagation of gradients in delta tuning is shown to be theoretically equivalent to co-state evolution in control theory, offering new guarantees and architectural insight for future delta design.

Comprehensive Empirical Study

Experiments are conducted on over 100 NLP tasks using T5 and RoBERTa backbones, yielding several critical findings:

• Performance: Although delta tuning does not surpass full fine-tuning in performance or convergence rates, the gap is modest (average accuracy within 2–5 points), and no single method predominates across tasks. The relation between tunable parameter count and performance is non-linear; architectural choices are more influential than raw parameter volume.
• Combinability: Joint application of multiple delta tuning methods is generally more effective, though optimal combinations vary by backbone, domain, and data regime. Combinability also introduces larger generalization gaps, indicating increased memorization even with reduced capacity.
• Scaling Law: All delta tuning methods demonstrate pronounced gains in performance and convergence as PLM size increases. For very large models (e.g., T5-XXL, 11B+ parameters), prompt tuning approaches full fine-tuning accuracy on SuperGLUE, and similar scaling phenomena are observed for adapter, LoRA, and prefix-based methods.
• Transferability: Tuned delta parameters exhibit non-trivial zero-shot transferability, particularly among tasks from similar domains (e.g., sentiment analysis, question answering), and can even support out-of-domain transfer in some generative scenarios.
• Efficiency: Delta tuning methods reduce peak GPU memory usage by up to 75% for small batch sizes, substantially lowering the barrier for large model experimentation. Time-to-convergence remains higher than full fine-tuning for small models, but this inefficiency diminishes with scaling.

Applications and Deployment

Delta tuning's practical advantages are highlighted in several domains:

• Fast Training and Shareable Checkpoints: Reduced storage and compute enable rapid experimentation and dissemination of adapted components, with open-source initiatives (AdapterHub, OpenDelta) supporting reproducible research and modular model composition.
• Multi-task Learning: Delta methods facilitate efficient multi-task and cross-lingual adaptation, supporting plug-and-play task specialization with minimal interference.
• Catastrophic Forgetting Mitigation: By confining updates to small parameter subsets, delta tuning is less prone to overwriting pretrained knowledge during continual or lifelong learning.
• Service-oriented Model Deployment: As APIs for large PLMs become standard, delta tuning offers parameter-efficient adaptation strategies for user-specific or multi-tenant models, supporting parallelized, privacy-preserving, or black-box tuning applications.

Implications and Future Directions

The findings have broad implications:

• Theory: The established low-rank and optimal control frameworks unify disparate delta tuning approaches and suggest new design principles for both architectural and algorithmic innovation.
• Practice: The consistent efficiency and transferability of delta methods point toward a future where large-scale PLMs can be rapidly repurposed for a wide range of tasks and user preferences without the prohibitive computational overhead of full fine-tuning.
• Society and Environment: Delta tuning may inherit social biases from underlying models, but its efficient, lightweight nature enables more accessible auditing and correction. Additionally, substantial reductions in compute and storage requirements can curtail the carbon footprint of model training and deployment.

Potential future work includes:

• Development of hybrid or dynamically composable delta architectures optimized for specific task distributions or deployment constraints.
• Theoretical exploration of the interplay between model overparameterization, intrinsic dimension, and transferability.
• Practical evaluation in real-world multitask and continual learning environments, including robust control-based adaptation designs.

Conclusion

This paper offers a rigorous, multifaceted examination of delta tuning, situating it as a central paradigm for efficient PLM adaptation. The synthesis of theoretical justification, empirical validation, and real-world deployment strategies establishes delta tuning as essential for scalable, flexible, and sustainable NLP with LLMs. The open-sourcing of tools and checkpoints further accelerates progress in the community toward modular, efficient, and interpretable model adaptation.