Meta Knowledge for Retrieval Augmented Large Language Models

Abstract: Retrieval Augmented Generation (RAG) is a technique used to augment LLMs with contextually relevant, time-critical, or domain-specific information without altering the underlying model parameters. However, constructing RAG systems that can effectively synthesize information from large and diverse set of documents remains a significant challenge. We introduce a novel data-centric RAG workflow for LLMs, transforming the traditional retrieve-then-read system into a more advanced prepare-then-rewrite-then-retrieve-then-read framework, to achieve higher domain expert-level understanding of the knowledge base. Our methodology relies on generating metadata and synthetic Questions and Answers (QA) for each document, as well as introducing the new concept of Meta Knowledge Summary (MK Summary) for metadata-based clusters of documents. The proposed innovations enable personalized user-query augmentation and in-depth information retrieval across the knowledge base. Our research makes two significant contributions: using LLMs as evaluators and employing new comparative performance metrics, we demonstrate that (1) using augmented queries with synthetic question matching significantly outperforms traditional RAG pipelines that rely on document chunking (p < 0.01), and (2) meta knowledge-augmented queries additionally significantly improve retrieval precision and recall, as well as the final answers breadth, depth, relevancy, and specificity. Our methodology is cost-effective, costing less than $20 per 2000 research papers using Claude 3 Haiku, and can be adapted with any fine-tuning of either the language or embedding models to further enhance the performance of end-to-end RAG pipelines.

• The paper introduces a PR3 framework that replaces traditional retrieve-then-read pipelines by generating synthetic QA pairs and metadata for improved query relevance.
• It demonstrates that augmented queries with Meta Knowledge Summaries yield significant improvements in recall, precision, and specificity (p < 0.01) over conventional methods.
• The study offers a cost-effective, scalable approach for enhancing domain-specific retrieval in LLMs, paving the way for future advances in automated metadata generation.

Meta Knowledge for Retrieval Augmented LLMs

Summary

This paper introduces an advanced methodology for Retrieval Augmented Generation (RAG) systems intended to enhance LLMs with domain-specific, time-critical, and contextually relevant information. The authors present a novel data-centric RAG workflow that transcends the traditional retrieve-then-read pipeline. They propose a prepare-then-rewrite-then-retrieve-then-read (PR3) framework, designed to achieve a higher level of domain-specific understanding by employing metadata and synthetic Question and Answer (QA) generation for each document. A new concept, Meta Knowledge Summary (MK Summary), is also introduced to augment and refine the user queries based on metadata clusters, resulting in more tailored and in-depth information retrieval.

Key Contributions

The paper makes significant methodological enhancements to RAG pipelines by embedding the following innovations:

1. Enhanced Workflow with PR3 Framework:
   • Transforms traditional retrieval methods into a more complex pipeline (prepare-then-rewrite-then-retrieve-then-read).
   • Prioritizes generating metadata and QA pairs to produce synthesized document-level understanding.
2. Introduction of Meta Knowledge Summary (MK Summary):
   • Employs metadata-generated clusters to create high-level summaries.
   • Facilitates tailored user-query augmentation to enhance retrieval precision, relevancy, breadth, and depth of final answers.
3. Performance Metrics and Cost Efficiency:
   • Demonstrates that augmented queries with synthetic question matching significantly outperform traditional RAG pipelines reliant on document chunking (statistical significance: $p < 0.01$).
   • Illustrates cost-effective processing, approximately \$20 per 2000 research papers utilizing Claude 3 Haiku.

Methodology Details

1. Synthetic QA Generation:
   • The authors utilize Chain of Thoughts (CoT) prompting with Claude 3 Haiku to create custom metadata and QA pairs.
   • Metadata defines document categories, leading to the generation of specific questions and answers that encapsulate the document's essential content.
2. Meta Knowledge Summary (MK Summary):
   • Meta knowledge is generated by compiling summaries of metadata-based document clusters using Claude 3 Sonnet.
   • MK Summary serves to dynamically augment user queries, enabling a richer, more focused search.
3. Augmented Query and Retrieval Process:
   • User queries are conditionally enhanced using metadata-driven MK Summary.
   • Synthetic QAs are embedded, replacing traditional document chunking in the vector space for retrieval.

Evaluation and Results

The evaluation methodology includes generating 200 synthetic user queries and comparing multiple retrieval strategies:

• Traditional document chunking.
• Document chunking combined with query augmentation.
• QA-based retrieval (without MK Summary).
• QA-based retrieval with MK Summary.

Six metrics—recall, precision, specificity, breadth, depth, and relevancy—are used to assess performance, using Claude 3 Sonnet as the evaluator. The results indicate substantial improvements across all metrics, with significant enhancements in breadth, depth, and specificity due to the introduction of MK Summary ($p < 0.01$).

Conclusion and Implications

The results underscore the efficacy of the PR3 framework and MK Summary in improving the performance and comprehension of RAG systems. The proposed methodology not only augments the retrieval accuracy and breadth of knowledge but also bolsters the depth and relevance of information provided by LLMs. The research aligns with ongoing efforts to reduce information loss inherent in document chunking and to facilitate complex, domain-specific reasoning within LLMs.

Future Directions

The implications of this research extend to various applications involving knowledge-intensive tasks requiring timely and relevant data integration. Future research may explore:

1. Enhanced Metadata Discovery:
   • Developing automated and iterative metadata generation techniques for more nuanced and comprehensive document categorization.
2. Multi-hop Iterative Searches:
   • Implementing iterative search and retrieval frameworks to further refine and deepen the synthesis of information from diverse document datasets.
3. Prompt Tuning for MK Summary:
   • Optimizing summary content for various domain-specific applications through prompt tuning and alternative summarization techniques.

By addressing these aspects, the proposed methodology sets a foundation for future advancements in autonomous, agent-based document database reasoning with LLMs, enhancing the utility and applicability of RAG systems across varied domains and applications.