Linguistically-Informed Multilingual Instruction Tuning: Is There an Optimal Set of Languages to Tune?

Abstract: Multilingual LLMs often perform unevenly across different languages due to limited generalization capabilities for some languages. This issue is significant because of the growing interest in making universal LLMs that work well for all languages. Instruction tuning with multilingual instruction-response pairs has been used to improve model performance across various languages. However, this approach is challenged by high computational costs, a lack of quality tuning data for all languages, and the "curse of multilinguality" -- the performance drop per language after adding many languages. Recent studies have found that working with datasets with few languages and a smaller number of instances can be beneficial. Yet, there exists no systematic investigation into how choosing different languages affects multilingual instruction tuning. Our study proposes a method to select languages for instruction tuning in a linguistically informed way, aiming to boost model performance across languages and tasks. We use a simple algorithm to choose diverse languages and test their effectiveness on various benchmarks and open-ended questions. Our results show that this careful selection generally leads to better outcomes than choosing languages at random. We suggest a new and simple way of enhancing multilingual models by selecting diverse languages based on linguistic features that could help develop better multilingual systems and guide dataset creation efforts. All resources, including the code for language selection and multilingual instruction tuning, are made available in our official repository at https://github.com/GGLAB-KU/ling-informed-mit enabling reproducibility and further research in this area.

• The paper introduces a linguistically informed approach that employs k‐means clustering to select optimal language subsets for multilingual instruction tuning.
• It demonstrates how leveraging geographical and typological features significantly enhances cross-lingual performance compared to random language selection.
• The research indicates that optimal language selection is model and task-dependent, offering practical guidelines to reduce computational costs and improve tuning outcomes.

Linguistically-Informed Multilingual Instruction Tuning: A Systematic Approach

This paper presents a study on enhancing multilingual instruction tuning (MIT) through a linguistically informed selection of languages, examining whether an optimal set of languages exists for tuning purposes. The challenge faced by multilingual LLMs in achieving uniform performance across various languages is a well-documented issue. This problem is exacerbated by the high computational costs, scarcity of quality tuning data, and the known "curse of multilinguality"—a performance decline as more languages are added. The authors aim to address these issues by introducing a systematized approach to language selection based on linguistic features.

Methodology

The authors propose using a k-means clustering algorithm to select diverse languages for instruction tuning. Different linguistically informed criteria are used, including typological feature vectors (TYPO), learned language vectors (LEARN), geographical features (GEO), and semantic typology (SEM). The study also considers random (RND) and language family (FAM) selections for comparison. By implementing these features, the authors aim to enhance cross-lingual and cross-task performance effectively.

Numerical Results and Analysis

The experiments conducted involve multiple model architectures, including mT5, mGPT, and BLOOM, across several multilingual benchmarks (XNLI, XCOPA, XStoryCloze, XWinograd, and PAWS-X). Notably, the GEO subset consistently showed enhanced performance across different models and tasks, often outperforming others. This indicates that geographical features might offer robust cross-lingual learning, suggesting a latent connection between geographical diversity and language modeling performance.

In terms of average performance, linguistically informed selections generally surpassed random baselines, though no singular method emerged as the definitive best across all benchmarks. This underscores the conclusion that optimal language selection is task and model-dependent. GEO and TYPO subsets provided considerable advantages, while all subsets, except random, improved over baseline models in unseen languages.

Implications and Future Directions

The implications of this research touch on both practical and theoretical domains. Practically, the findings guide multilingual LLM training towards more efficient language subset selection, potentially reducing computational resources without sacrificing performance. Theoretically, the study lays groundwork for exploring intrinsic language relationships within a multi-language context, promoting further research on linguistic diversity's role in model performance.

Future developments may include exploring alternative clustering algorithms or combining linguistic features with other informative data. Expanding the model sizes and types could further reveal scaling behaviors, while also examining real-world applications and evaluating against human-centered metrics can bridge any gaps between theoretical advancements and practical applicability.

In conclusion, this study contributes significant insights into MIT by systematically leveraging linguistic diversity for enhancing LLM performance. While challenges remain, such structured explorations move the field closer toward universal LLM efficacy.