Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs

Abstract: Learning on Graphs has attracted immense attention due to its wide real-world applications. The most popular pipeline for learning on graphs with textual node attributes primarily relies on Graph Neural Networks (GNNs), and utilizes shallow text embedding as initial node representations, which has limitations in general knowledge and profound semantic understanding. In recent years, LLMs have been proven to possess extensive common knowledge and powerful semantic comprehension abilities that have revolutionized existing workflows to handle text data. In this paper, we aim to explore the potential of LLMs in graph machine learning, especially the node classification task, and investigate two possible pipelines: LLMs-as-Enhancers and LLMs-as-Predictors. The former leverages LLMs to enhance nodes' text attributes with their massive knowledge and then generate predictions through GNNs. The latter attempts to directly employ LLMs as standalone predictors. We conduct comprehensive and systematical studies on these two pipelines under various settings. From comprehensive empirical results, we make original observations and find new insights that open new possibilities and suggest promising directions to leverage LLMs for learning on graphs. Our codes and datasets are available at https://github.com/CurryTang/Graph-LLM.

• The paper proposes two pipelines (LLMs-as-Enhancers and LLMs-as-Predictors) to integrate LLM capabilities into graph-based tasks, enhancing node feature representation and prediction.
• Experiments reveal that feature-level enhancements using cascading and iterative strategies substantially boost GNN performance, although low-label scenarios pose challenges.
• The study highlights future directions such as self-training and advanced prompting to better incorporate structural graph context and mitigate dataset biases.

Exploring the Potential of LLMs in Learning on Graphs

Introduction

The integration of LLMs with Graph Neural Networks (GNNs) presents a promising avenue for enhancing the semantic understanding and knowledge incorporation in graph-based machine learning tasks. This paper investigates two primary pipelines for leveraging LLMs in graph scenarios: LLMs-as-Enhancers and LLMs-as-Predictors. LLMs-as-Enhancers utilize LLMs to enrich the text attributes on nodes before GNNs make final predictions. LLMs-as-Predictors are tasked with making predictions directly, treating graph-based tasks similarly to text-based tasks handled by LLMs.

LLMs as Enhancers

The LLMs-as-Enhancers pipeline proposes using LLMs to enhance textual attributes of nodes on graphs. This approach is rooted in leveraging the contextual understanding of LLMs which could alleviate limitations associated with traditional shallow text embeddings used in GNNs.

Figure 1: An illustration of LLMs-as-Enhancers, where LLMs pre-process the text attributes, and GNNs eventually make the predictions.

Strategies for Enhancing Text Attributes

Figure 2: Three strategies to adopt LLMs as enhancers. The first two integrating structures are designed for feature-level enhancement, while the last structure is designed for text-level enhancement.

1. Feature-Level Enhancement: This approach uses the embeddings generated by embedding-visible LLMs to numerically represent node text attributes, which are then processed by GNNs. Two integrating structures are explored:
   • Cascading Structure: Embedding-visible LLMs and GNNs are combined sequentially where LLMs encode text which GNNs use as input features.
   • Iterative Structure: Co-training of GNNs and PLMs allows mutual improvement through iterative pseudo-label generation.
2. Text-Level Enhancement: Here, embedding-invisible LLMs are tasked with generating augmented textual attributes. The original and augmented textual attributes are ensembled to form a robust set of node features.

Performance and Scalability

The experiments revealed that deep sentence embedding models such as Sentence-BERT and e5-large, when combined with GNNs, achieve robust performance with efficiency across various datasets. However, caution is needed in low-label scenarios, as seen where models relying heavily on PLM fine-tuning showed decreased efficacy.

LLMs as Predictors

The LLMs-as-Predictors paradigm evaluates the capability of LLMs to perform node classification by treating it as a text classification problem, excluding graph structural information.

Figure 3: Illustrations for TAPE and KEA. TAPE leverages the knowledge of LLMs to generate explanations for their predictions.

Direct Prediction and Structure Incorporation

Initial results demonstrated encouraging zero-shot performance by LLMs on text rich datasets, aligning closely with traditional classifiers on datasets with clearly annotated labels. However, the outputs often showed reasonable yet incorrect predictions compared to the annotated ground truths, revealing potential biases in existing datasets rather than LLM capability limitations. Incorporating structural information via prompts remains under-explored for enhanced alignment with graph structures.

Limitations and Future Work

Future work should explore self-training methods to utilize LLM-generated pseudo-labels efficiently, investigate the use of more advanced prompting strategies for better aligning graph structure information with LLM capabilities, and handle dataset annotations more effectively to mitigate the observed bias and noise in existing labels.

Conclusion

This paper reveals promising avenues for integrating LLMs into graph learning tasks and highlights potential pitfalls in evaluation and scalability. With further refinements, especially in context alignment and efficiency, LLMs could significantly broaden the horizon for graph-based machine learning applications.