Identifying Task Groupings for Multi-Task Learning Using Pointwise V-Usable Information

Abstract: The success of multi-task learning can depend heavily on which tasks are grouped together. Naively grouping all tasks or a random set of tasks can result in negative transfer, with the multi-task models performing worse than single-task models. Though many efforts have been made to identify task groupings and to measure the relatedness among different tasks, it remains a challenging research topic to define a metric to identify the best task grouping out of a pool of many potential task combinations. We propose a metric of task relatedness based on task difficulty measured by pointwise V-usable information (PVI). PVI is a recently proposed metric to estimate how much usable information a dataset contains given a model. We hypothesize that tasks with not statistically different PVI estimates are similar enough to benefit from the joint learning process. We conduct comprehensive experiments to evaluate the feasibility of this metric for task grouping on 15 NLP datasets in the general, biomedical, and clinical domains. We compare the results of the joint learners against single learners, existing baseline methods, and recent LLMs, including Llama 2 and GPT-4. The results show that by grouping tasks with similar PVI estimates, the joint learners yielded competitive results with fewer total parameters, with consistent performance across domains.

• The paper demonstrates that grouping tasks with similar pvi estimates effectively mitigates negative transfer in multi-task learning.
• The proposed two-stage method calculates pvi using pre-trained models and groups tasks based on statistical similarity for enhanced performance.
• Empirical evaluations across diverse NLP domains show that pvi-based groupings improve efficiency and reduce parameter tuning compared to traditional methods.

Identifying Task Groupings for Multi-Task Learning Using Pointwise $\mathcal{V}$-Usable Information

The paper "Identifying Task Groupings for Multi-Task Learning Using Pointwise $\mathcal{V}$-Usable Information" presents a novel approach to optimizing task groupings in multi-task learning (MTL) through a metric known as pointwise $\mathcal{V}$-usable information (pvi). The central theme of this research is to address the challenge of negative transfer in MTL—where naive task groupings may perform worse than single-task models—by utilizing task difficulty estimates provided by pvi to identify beneficial task combinations.

In traditional MTL, grouping tasks without careful consideration can result in negative transfer, leading to sub-optimal performance when compared to single-task learning (STL). Despite various efforts to define task relatedness and optimize task combinations, identifying the most effective task groupings remains an open research question. This study hypothesizes that tasks with similar pvi estimates could enhance MTL performance by ensuring tasks with comparable difficulty levels are learned together.

The proposed method involves a two-stage process: first, calculating pvi estimates for each task using a pre-trained model by assessing the usable information each dataset provides; second, grouping tasks based on the statistical similarity of their pvi distributions. This approach is evaluated across 15 NLP datasets spanning general, biomedical, and clinical domains, using models such as roberta-large and Bio+Clinical BERT.

Key findings highlight that by grouping tasks with comparable pvi distributions, MTL models perform competitively or better than single-task models, especially regarding efficiency in parameter tuning and reducing overfitting. The results also show consistent performance across domains with fewer total parameters, indicating the efficacy of the proposed method in practical MTL applications.

The research further compares its approach with state-of-the-art task grouping methods, such as task embedding and surrogate models. Empirically, pvi-based groupings proved advantageous, often surpassing alternative methods in improving task performance in MTL settings.

Additionally, the study investigates the performance of LLMs such as Llama 2 and GPT-4 using few-shot prompting. Although LLMs demonstrate substantial capabilities in several tasks, fine-tuned domain-specific models—both single and multi-task—consistently surpass LLM performance in specialized tasks, particularly in biomedical and clinical domains.

This research offers significant implications for MTL, suggesting that task difficulty measured via pvi can serve as a robust metric for task relatedness, guiding the discovery of effective task groupings and mitigating negative transfer effects. Practically, this method provides a systematic and computationally efficient way to leverage domain-specific models, enhancing their generalization across multiple tasks.

Theoretically, the use of pvi provides a quantitative basis for understanding task similarities and dependencies, offering a new dimension to task selection in MTL. Future research could explore adapting this framework to instance selection within datasets and further optimize task-specific parameter sharing strategies in MTL architectures.

In conclusion, the utilization of pointwise $\mathcal{V}$-usable information offers a promising avenue in advancing the discipline of multi-task learning, particularly in selecting task groupings that facilitate positive transfer and improved learning efficiencies across varied and complex data domains.