LERT: A Linguistically-motivated Pre-trained Language Model

Abstract: Pre-trained LLM (PLM) has become a representative foundation model in the natural language processing field. Most PLMs are trained with linguistic-agnostic pre-training tasks on the surface form of the text, such as the masked LLM (MLM). To further empower the PLMs with richer linguistic features, in this paper, we aim to propose a simple but effective way to learn linguistic features for pre-trained LLMs. We propose LERT, a pre-trained LLM that is trained on three types of linguistic features along with the original MLM pre-training task, using a linguistically-informed pre-training (LIP) strategy. We carried out extensive experiments on ten Chinese NLU tasks, and the experimental results show that LERT could bring significant improvements over various comparable baselines. Furthermore, we also conduct analytical experiments in various linguistic aspects, and the results prove that the design of LERT is valid and effective. Resources are available at https://github.com/ymcui/LERT

• The paper introduces a linguistically-informed pre-training framework that integrates POS, NER, and dependency parsing tasks into the masked language model.
• The model employs a pre-training warmup strategy to assimilate basic linguistic features faster, enhancing performance on complex tasks like MRC and NER.
• Comprehensive evaluations across ten Chinese NLU tasks demonstrate that structured linguistic knowledge significantly improves semantic representations and task outcomes.

Overview of "LERT: A Linguistically-motivated Pre-trained LLM"

The paper "LERT: A Linguistically-motivated Pre-trained LLM" introduces LERT, a pre-trained LLM (PLM) that integrates linguistic features into the masked LLM (MLM) pre-training task, utilizing a linguistically-informed pre-training (LIP) strategy. This approach aims to imbue the PLM with richer linguistic features, addressing the limitations of existing methods that predominantly rely on surface form-based and linguistic-agnostic pre-training tasks.

Key Contributions

1. Linguistically-informed Pre-training: LERT incorporates three distinct linguistic tasks—part-of-speech (POS) tagging, named entity recognition (NER), and dependency parsing (DEP)—into its training regime alongside the traditional MLM task. This multi-task pre-training scheme demonstrates a novel methodology for embedding linguistic knowledge into PLMs.
2. Linguistically-informed Pre-training (LIP) Strategy: To optimize the learning dynamics, LERT employs a LIP strategy that accelerates foundational linguistic tasks faster than more complex ones. This approach aids in gradually enriching the model's linguistic understanding, potentially mirroring human cognitive processes where basic knowledge is acquired prior to more complex constructs.
3. Comprehensive Evaluation: The authors conducted extensive experiments across ten Chinese natural language understanding (NLU) tasks. LERT showed significant performance gains over competitive baselines, particularly excelling in complex tasks such as machine reading comprehension (MRC) and named entity recognition (NER).

Methodological Insights

LERT is trained on conventional Chinese corpora with a specific emphasis on integrating linguistic features through weakly-supervised data derived from linguistic processing tools like LTP. The architecture of LERT involves a multi-task setup where the contributions of individual linguistic tasks are analyzed thoroughly. This setup not only allows LERT to outperform baselines on NLU tasks but also highlights the importance of incorporating structured linguistic information into PLMs.

A distinctive aspect of LERT’s design is the pre-training warmup strategy, ensuring that fundamental linguistic knowledge (e.g., POS) is assimilated faster than complex knowledge (e.g., DEP). This nuanced task ordering significantly enhanced LERT’s performance, particularly on NLU tasks that implicitly require linguistic insight.

Results and Implications

The empirical results affirm that the integration of linguistic tasks in the pre-training phase leads to improved performance across numerous NLP tasks. Noteworthy is LERT’s superior performance in tasks necessitating deeper linguistic comprehension—MRC and NER emerge as beneficiaries of this design. While enhancements on simpler tasks like text classification were less dramatic, the general trend supports the notion that linguistic features contribute to more refined semantic representations.

Furthermore, the model's architecture and training regime present a scalable framework for future endeavors to incorporate additional types of linguistic knowledge. The semantic depth capacity of LERT could feasibly extend to other languages and domains, thus broadening its utility in diverse NLP applications.

Future Directions

The proven efficacy of the linguistically-informed pre-training strategy suggests avenues for further exploration, such as incorporating advanced semantic features or experimenting with dynamic task weighting throughout training. The findings also prompt consideration of similar strategies in other PLM designs, potentially guiding innovations in multilingual or domain-specific LLMs.

In conclusion, LERT stands as a credible advancement in the evolution of PLMs, utilizing linguistic features in a sophisticated training regimen. Its robust evaluation and innovative integration strategies offer a template for amplifying linguistic insight within NLP frameworks, which remains a pivotal component in the ongoing development of nuanced, context-aware LLMs.