Retrieval-Augmented Generation with Hierarchical Knowledge

Abstract: Graph-based Retrieval-Augmented Generation (RAG) methods have significantly enhanced the performance of LLMs in domain-specific tasks. However, existing RAG methods do not adequately utilize the naturally inherent hierarchical knowledge in human cognition, which limits the capabilities of RAG systems. In this paper, we introduce a new RAG approach, called HiRAG, which utilizes hierarchical knowledge to enhance the semantic understanding and structure capturing capabilities of RAG systems in the indexing and retrieval processes. Our extensive experiments demonstrate that HiRAG achieves significant performance improvements over the state-of-the-art baseline methods.

• The paper introduces HiRAG, which overcomes limitations in flat RAG systems by integrating hierarchical knowledge for improved semantic coherence.
• It employs a two-part framework featuring hierarchical indexing and retrieval to bridge local and global information gaps.
• Experimental results show that HiRAG outperforms methods like GraphRAG in generating more accurate and diverse responses.

Retrieval-Augmented Generation with Hierarchical Knowledge

Introduction

The paper "Retrieval-Augmented Generation with Hierarchical Knowledge" (2503.10150) introduces HiRAG, an innovative approach to augmenting LLMs with hierarchical knowledge for enhanced retrieval-augmented generation (RAG) systems. RAG systems aim to improve LLMs' performance in domain-specific tasks by utilizing external data during both indexing and retrieval phases. HiRAG addresses significant limitations in existing graph-structured RAG systems, particularly the distant structural relations between semantically similar entities and the knowledge gap between local and global information representations (Figure 1).

Figure 1: The challenges faced by existing RAG systems: (1) Distant structural relationship between semantically similar entities. (2) Knowledge gap between local and global knowledge.

The HiRAG Framework

HiRAG is structured around two core components: hierarchical indexing (HiIndex) and hierarchical retrieval (HiRetrieval), both leveraging a hierarchical knowledge graph (KG) architecture for improved semantic and relational coherence (Figure 2).

Figure 2: The overall architecture of the HiRAG framework.

Indexing with Hierarchical Knowledge

HiIndex constructs a hierarchical KG by employing entity-centric triple extraction followed by semantical clustering using Gaussian Mixture Models. This process builds multi-layer knowledge graphs where entities from lower layers are summarized at higher levels, enhancing connectivity between semantically similar entities that may be structurally distant in a flat graph setting. The hierarchical approach enables the system to bridge semantic gaps through summarized higher-layer entities, addressing the deficiencies noted in other RAG systems (Figure 3).

Figure 3: Comparisons between the clustering coefficients of GraphRAG, LightRAG and HiRAG across four datasets.

Retrieval with Hierarchical Knowledge

HiRetrieval integrates hierarchical knowledge to capture local, global, and bridge-level contexts for query-response generation. By employing reasoning paths between retrieved communities, HiRetrieval effectively aligns local entity descriptions with broader community insights, resolving the knowledge disparity issue. This structured retrieval process allows the generation module (LLM) to utilize comprehensive context, informed by both granular and holistic knowledge perspectives, improving response accuracy and coherence (Figure 4).

Figure 4: The shortest path with hierarchical KG between the entities in the case mentioned in the introduction.

Experimental Evaluation

Extensive experiments validate HiRAG's performance across diverse datasets against state-of-the-art RAG methods like GraphRAG, LightRAG, and KAG. Key findings indicate that HiRAG consistently outperforms these methods in comprehensiveness, empowerment, and diversity of generated responses by leveraging its hierarchical knowledge integration. HiRAG's architecture significantly enhances LLM reasoning capabilities, mitigating the limitations of both flat-graph RAG systems and traditional local-global knowledge representations.

Implications and Future Directions

The hierarchical design ensures that HiRAG effectively balances structural and semantic connectivity, paving the way for more intuitive and accurate LLM responses in complex retrieval-augmented tasks. This framework opens new avenues for incorporating hierarchical knowledge in AI systems, suggesting further exploration into scalable indexing methods and more sophisticated query-aware ranking mechanisms to enhance retrieval quality.

Future research could focus on optimizing HiRAG's algorithms for faster indexing, perhaps through parallel processing, and integrating causality into hierarchical frameworks to better support long-term inferencing and reasoning in LLMs.

Conclusion

HiRAG represents a significant advancement in retrieval-augmented generation methodologies, with its hierarchical approach overcoming crucial limitations in existing systems. By integrating unsupervised hierarchical indexing and bridging mechanisms, it offers a robust solution for nuanced, context-aware knowledge generation, setting the precedent for next-generation RAG systems.