What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition

Abstract: The knowledge encapsulated in a model is the core factor determining its final performance on downstream tasks. Much research in NLP has focused on efficient methods for storing and adapting different types of knowledge, e.g., in dedicated modularized structures, and on how to effectively combine these, e.g., by learning additional parameters. However, given the many possible options, a thorough understanding of the mechanisms involved in these compositions is missing, and hence it remains unclear which strategies to utilize. To address this research gap, we propose a novel framework for zero-shot module composition, which encompasses existing and some novel variations for selecting, weighting, and combining parameter modules under a single unified notion. Focusing on the scenario of domain knowledge and adapter layers, our framework provides a systematic unification of concepts, allowing us to conduct the first comprehensive benchmarking study of various zero-shot knowledge composition strategies. In particular, we test two module combination methods and five selection and weighting strategies for their effectiveness and efficiency in an extensive experimental setup. Our results highlight the efficacy of ensembling but also hint at the power of simple though often-ignored weighting methods. Further in-depth analyses allow us to understand the role of weighting vs. top-k selection, and show that, to a certain extent, the performance of adapter composition can even be predicted.

• The paper introduces a unified framework for zero-shot knowledge composition that utilizes adapter layers to select, weight, and combine domain-specific knowledge without additional training.
• It benchmarks five adapter weighting strategies across 21 training and 10 evaluation domains, with ensembling consistently outperforming parameter averaging.
• The research provides meta-regression analysis and publicly available resources, ensuring reproducibility and guiding future investigations in knowledge modularization and domain adaptation.

Introduction

Pre-trained LLMs (PLMs) such as GPT and BERT have dramatically advanced the field of NLP, which can be attributed to the vast amount of knowledge encapsulated within their parameters. In pursuit of optimizing the use of PLMs for domain-specific tasks, a considerable amount of research has focused on strategies for knowledge modularization and composition. Particularly in zero-shot settings, the goal is to leverage and combine knowledge from various pre-trained modules to improve performance on target domains without additional training.

A Unified Composition Framework

The paper introduces a novel, comprehensive framework for zero-shot knowledge composition, applicable across various scenarios and modular structures. It centers around the concept of adapter layers for domain adaptation and unfolds in three conceptual steps: selecting relevant adapters, weighting them, and performing the final combination. The paper details five unique scoring strategies for adapter selection and weighting: Uniform, Semantic Sentence Similarity, TF–IDF, Domain Prior, and Entropy. Utilizing these strategies, the framework tests combinations through both parameter averaging, akin to "model souping," and output vector ensembling.

Benchmarking Composition Strategies

Extensive experiments across 21 training and 10 evaluation domains, involving three distinct models (gpt2-base, gpt2-large, deberta-base), benchmark the composition strategies for zero-shot domain adaptation. Results demonstrate that ensembling typically surpasses parameter averaging regarding effectiveness. Furthermore, against expectations, corpus-based strategies such as tf–idf and sentence similarity often outperformed more complex model-based approaches in adapter weighting and provided greater efficiency. Meta-regression analysis was also conducted to predict the performance of adapter combinations on unseen domains, proving partially successful especially for specific adapter compositions.

Related Work and Reproducibility

This study builds on extensive literature concerning knowledge modularization and composition, differentiating itself by providing a unified framework and analysis across various methods. The paper's experimental settings and resources are detailed to ensure reproducibility. For complete transparency and to support further research, the authors have made the code and models publicly available.

Conclusion

In summary, this research presents a unified approach to zero-shot knowledge composition with a detailed benchmarking study to evaluate various strategies. It highlights the efficacy of ensembling over parameter averaging and the surprising effectiveness and efficiency of simplistic adapter weighting techniques. Through meta-regression, it also opens avenues for predicting the performance of domain adaptation methods, streamlining future explorations. With its publication, the authors encourage further investigations into effective knowledge composition, aiming to further enhance the adaptability and efficiency of NLP systems.