Making Large Language Models Perform Better in Knowledge Graph Completion

Abstract: LLM based knowledge graph completion (KGC) aims to predict the missing triples in the KGs with LLMs. However, research about LLM-based KGC fails to sufficiently harness LLMs' inference proficiencies, overlooking critical structural information integral to KGs. In this paper, we explore methods to incorporate structural information into the LLMs, with the overarching goal of facilitating structure-aware reasoning. We first discuss on the existing LLM paradigms like in-context learning and instruction tuning, proposing basic structural information injection approaches. Then we propose a Knowledge Prefix Adapter (KoPA) to fulfill this stated goal. The KoPA uses a structural pre-training phase to comprehend the intricate entities and relations within KGs, representing them as structural embeddings. Then KoPA communicates such cross-modal structural information understanding to the LLMs through a knowledge prefix adapter which projects the structural embeddings into the textual space and obtains virtual knowledge tokens positioned as a prefix of the input prompt. We conduct comprehensive experiments and provide incisive analysis concerning how the introduction of cross-modal structural information would be better for LLM's factual knowledge reasoning ability. Our code and data are available at https://github.com/zjukg/KoPA .

• The paper presents KoPA, a novel adapter that infuses structural KG data into LLMs through structural pre-training and cross-modal projection.
• The methodology extends standard LLM paradigms by converting structural embeddings into virtual tokens, yielding higher accuracy and F1 scores on benchmarks like CoDeX-S.
• The study demonstrates KoPA’s strong transferability to unseen KG entities while preserving LLMs’ overall language abilities, pointing to future multi-modal AI advancements.

Enhancing LLMs for Knowledge Graph Completion

The paper "Making LLMs Perform Better in Knowledge Graph Completion" introduces innovative methodologies to improve the capabilities of LLMs in the task of Knowledge Graph Completion (KGC). The primary focus is on integrating structural information from Knowledge Graphs (KGs) into LLMs, addressing the prevalent issue where LLMs often fail to leverage the intricate structural data critical for KGC.

The authors identify a gap in existing research where LLM-based KGC approaches do not fully exploit the inference capabilities of LLMs when structural knowledge from KGs is overlooked. To bridge this gap, they propose a novel Knowledge Prefix Adapter (KoPA), designed to enable structure-aware reasoning within LLMs. This mechanism incorporates a structural pre-training phase that generates structural embeddings for the entities and relations in a KG. These embeddings are then projected into the textual space of the LLM, effectively creating "virtual knowledge tokens" that serve as a prefix to the input prompt.

Methodological Advancements

1. Extending LLM Paradigms:
   • Existing paradigms like in-context learning and instruction tuning have been extended by embedding structural information directly into LLM prompts. This involves leveraging basic structural data injection techniques which enrich the context available to the LLMs.
2. Knowledge Prefix Adapter (KoPA):
   • KoPA involves two critical stages:
     • Structural Pre-training: Uses self-supervised learning to capture the structural characteristics of KGs into embeddings.
     • Cross-modal Projection: Translates these structural embeddings into LLM-compatible knowledge tokens, facilitating their utilization in textual input prompts.
   • This approach enhances LLM performance by allowing it to consider structural nuances during reasoning processes.
3. Comprehensive Evaluation:
   • The paper evaluates the proposed approaches on several benchmark datasets (UMLS, CoDeX-S, FB15K-237N) demonstrating significant improvements over baseline models, including traditional embedding-based and PLM-based KGC methods.

Results and Insights

The empirical results illustrate that KoPA outperforms existing techniques by effectively merging textual and structural knowledge, achieving higher accuracy and F1 scores in triple classification tasks. For example, on the CoDeX-S dataset, KoPA demonstrates a performance uplift with an 82.74% accuracy and an 84.11% F1 score, surpassing traditional and PLM approaches.

Additionally, the authors conduct transferability and common ability retention experiments:

• Transferability: Highlights KoPA's ability to generalize to unseen entities, maintaining performance under inductive settings where parts of the KG not observed during training are included during testing.
• Common Ability Retention: Assesses KoPA’s impact on the general capabilities of LLMs using the MMLU benchmark to ensure that task-specific fine-tuning does not degrade overall language understanding and generation capabilities.

Implications and Future Directions

This research provides key insights into enhancing KGC tasks via LLMs by incorporating graph structural data, thereby making LLMs not only serve as robust language processing models but also as effective reasoning tools over structured data.

The implications of the study stretch beyond KGC, suggesting future pathways for developing multi-modal LLMs that robustly support various data types, including graphs. It also suggests exploring more sophisticated adapters for even higher-dimensional transformations across different modalities, which may lead to further advancements in AI-driven knowledge management systems and applications.

In conclusion, the innovative integration of KG structural information into LLMs marks a significant step forward in the evolution of AI systems, promoting the potential to handle complex reasoning tasks with improved fidelity and reliability. This invites further exploration into unified frameworks that harness both textual and structured data for enhanced AI applications across diverse domains.