On Transferability of Prompt Tuning for Natural Language Processing

Abstract: Prompt tuning (PT) is a promising parameter-efficient method to utilize extremely large pre-trained LLMs (PLMs), which can achieve comparable performance to full-parameter fine-tuning by only tuning a few soft prompts. However, PT requires much more training time than fine-tuning. Intuitively, knowledge transfer can help to improve the efficiency. To explore whether we can improve PT via prompt transfer, we empirically investigate the transferability of soft prompts across different downstream tasks and PLMs in this work. We find that (1) in zero-shot setting, trained soft prompts can effectively transfer to similar tasks on the same PLM and also to other PLMs with a cross-model projector trained on similar tasks; (2) when used as initialization, trained soft prompts of similar tasks and projected prompts of other PLMs can significantly accelerate training and also improve the performance of PT. Moreover, to explore what decides prompt transferability, we investigate various transferability indicators and find that the overlapping rate of activated neurons strongly reflects the transferability, which suggests how the prompts stimulate PLMs is essential. Our findings show that prompt transfer is promising for improving PT, and further research shall focus more on prompts' stimulation to PLMs. The source code can be obtained from https://github.com/thunlp/Prompt-Transferability.

• The paper demonstrates that prompt tuning achieves comparable performance to full fine-tuning using fewer parameters, despite requiring longer training time.
• It reveals that soft prompts can be effectively transferred in zero-shot and cross-model settings to enhance performance on similar tasks.
• The study identifies overlapping activated neuron rates as a critical indicator of successful prompt transfer, validated across 17 NLP tasks.

Exploring the Transferability of Prompt Tuning in NLP

The paper "On Transferability of Prompt Tuning for Natural Language Processing" conducts a comprehensive empirical analysis of prompt tuning (PT) as a parameter-efficient method to enhance the use and effectiveness of large pre-trained LLMs (PLMs). With the increasing size of PLMs, finding efficient tuning methods becomes crucial, and PT offers a solution by adjusting only a small number of learnable soft prompts instead of the full PLM parameters.

Key Findings and Contributions

1. Efficiency Trade-off: The paper identifies that while PT can achieve comparable performance to full fine-tuning with significantly fewer parameters, it often requires more training time to reach convergence. This indicates a trade-off between parameter efficiency and training time that the paper aims to address through knowledge transfer.
2. Zero-shot and Initialization Transfer: The research explores the transferability of soft prompts across different tasks and PLMs. It is observed that in zero-shot settings, soft prompts perform effectively when transferred to tasks of similar nature on the same PLM. Furthermore, when transferred across different PLMs using a cross-model projector on related tasks, they retain utility. These findings underscore the potential of prompt transfer in enhancing training efficiency and task performance.
3. Transferability Indicators: A novel aspect of the study is the investigation into what determines prompt transferability. It highlights the overlapping rate of activated neurons as a strong indicator of successful transfer. This suggests that understanding how prompts stimulate PLMs at a neural level is essential for improving transferability.
4. Experimental Validation: The efficacy of cross-task and cross-model prompt transfer is validated on 17 NLP tasks across 6 task types, using PLM series such as RoBERTa and T5. The paper reports significant improvements in training speeds and task performances when using transferable prompt tuning with initialization strategies.
5. Implications for Future Research: By showing that transferable methods can enhance PT efficiency, the study opens avenues for further research into optimizing prompt stimulation in PLMs and designing more generalized projector models for cross-model transfers.

Practical Applications

The paper paves the way for more efficient use of PLMs in practical applications by leveraging the transferability of prompt tuning. As PLMs become integral in various NLP tasks, improving PT efficiency will have substantial implications for computational costs and resource allocation in developing AI systems.

Theoretical Implications

On a theoretical front, the work prompts further exploration into neural activation patterns and their role in knowledge transfer within deep learning models. The findings encourage a deeper dive into the structural properties of PLMs that facilitate prompt efficacy and transferability.

In conclusion, the paper provides robust empirical support for the potential of prompt transfer methods to enhance the efficiency of PT and offers valuable insights into the underlying mechanisms that govern transferability in large-scale LLMs. These contributions are likely to influence the development of future adaptive and resource-efficient NLP systems.