HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction

Abstract: Extraction and interpretation of intricate information from unstructured text data arising in financial applications, such as earnings call transcripts, present substantial challenges to LLMs even using the current best practices to use Retrieval Augmented Generation (RAG) (referred to as VectorRAG techniques which utilize vector databases for information retrieval) due to challenges such as domain specific terminology and complex formats of the documents. We introduce a novel approach based on a combination, called HybridRAG, of the Knowledge Graphs (KGs) based RAG techniques (called GraphRAG) and VectorRAG techniques to enhance question-answer (Q&A) systems for information extraction from financial documents that is shown to be capable of generating accurate and contextually relevant answers. Using experiments on a set of financial earning call transcripts documents which come in the form of Q&A format, and hence provide a natural set of pairs of ground-truth Q&As, we show that HybridRAG which retrieves context from both vector database and KG outperforms both traditional VectorRAG and GraphRAG individually when evaluated at both the retrieval and generation stages in terms of retrieval accuracy and answer generation. The proposed technique has applications beyond the financial domain

• The paper demonstrates that HybridRAG combines vector retrieval and knowledge graphs for enhanced extraction of complex financial information.
• It employs advanced techniques to integrate contextual document chunks with structured entity relationships, boosting answer relevancy.
• Evaluation metrics show improved faithfulness and response relevance, asserting the method's utility for financial data analysis.

HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction

Introduction

The paper "HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction" (2408.04948) addresses the challenge of extracting and interpreting intricate information from unstructured text data in financial domains. Typical LLMs, including those leveraging Retrieval-Augmented Generation (RAG) techniques that utilize vector databases, encounter obstacles such as domain-specific terminology and complex document formats when applied to financial applications. This study introduces HybridRAG, a novel approach that combines Knowledge Graphs (KGs) with vector-based RAG techniques to enhance question-answer (Q&A) systems for financial document information extraction. The findings indicate that HybridRAG outperforms traditional VectorRAG and GraphRAG individually in terms of retrieval accuracy and answer generation.

Methodology

The methodology encompasses the integration of two primary methods:

VectorRAG: This approach involves dividing external documents into chunks, transforming these into embeddings using a model like OpenAI's text-embedding-ada-002, and storing them in a vector database. The RAG process begins with formulating a query to search this database and retrieve relevant document chunks, which are then used to provide context to LLMs, enhancing response relevance and coherence.

Figure 1: A schematic diagram describing the vector database creation of a RAG application.

Knowledge Graph Construction: KGs represent entities and their relationships in a structured form, which is advantageous in financial contexts for capturing domain-specific insights. Methods for building KGs involve knowledge extraction (identifying entities and relationships), knowledge improvement (removing redundancies and filling gaps), and using algorithms for efficient querying. These graphs are then utilized in GraphRAG to encode structured information that LLMs can interpret, feeding this into response generation.

Figure 2: A schematic diagram describing knowledge graph creation process of GraphRAG.

HybridRAG Technique: The proposed approach integrates VectorRAG and GraphRAG by combining contextual information retrieved from both systems, providing a comprehensive and enriched input to an LLM. This hybrid methodology demonstrates superior capabilities in generating contextually accurate responses, surpassing the limitations of using either VectorRAG or GraphRAG in isolation.

Evaluation Metrics and Results

The study employs multiple metrics to evaluate retrieval and generation performance, including faithfulness, answer relevance, and context relevance. The evaluation highlighted that:

• Faithfulness scores showed that HybridRAG maintained a high level of factual consistency with contextually retrieved information.
• Answer Relevance was notably high for HybridRAG, indicating its efficacy in generating pertinent responses.
• Context Precision and Recall metrics revealed that although HybridRAG had slightly lower precision due to the fused context approach, it excelled in recalling comprehensive context.

These results affirm that HybridRAG provides a balanced performance over existing RAG methodologies, offering improvements specifically crucial in the financial domain.

Conclusion

HybridRAG marks a significant advancement in the automation of information extraction from complex financial documents. By merging the structural benefits of KGs and the contextual depth of vector-based RAG models, this approach ensures higher retrieval accuracy and improved answer generation. The implications of such advancements are extensive, potentially transforming financial analysis by providing tools that facilitate better data-driven decision-making. Future work may encompass expanding this approach to integrate real-time financial data and numerical analysis capabilities, thus broadening its applicability across dynamic business environments.