Parametric Retrieval Augmented Generation

Abstract: Retrieval-augmented generation (RAG) techniques have emerged as a promising solution to enhance the reliability of LLMs by addressing issues like hallucinations, outdated knowledge, and domain adaptation. In particular, existing RAG methods append relevant documents retrieved from external corpus or databases to the input of LLMs to guide their generation process, which we refer to as the in-context knowledge injection method. While this approach is simple and often effective, it has inherent limitations. Firstly, increasing the context length and number of relevant documents can lead to higher computational overhead and degraded performance, especially in complex reasoning tasks. More importantly, in-context knowledge injection operates primarily at the input level, but LLMs store their internal knowledge in their parameters. This gap fundamentally limits the capacity of in-context methods. To this end, we introduce Parametric retrieval-augmented generation (Parametric RAG), a new RAG paradigm that integrates external knowledge directly into the parameters of feed-forward networks (FFN) of an LLM through document parameterization. This approach not only saves online computational costs by eliminating the need to inject multiple documents into the LLMs' input context, but also deepens the integration of external knowledge into the parametric knowledge space of the LLM. Experimental results demonstrate that Parametric RAG substantially enhances both the effectiveness and efficiency of knowledge augmentation in LLMs. Also, it can be combined with in-context RAG methods to achieve even better performance. We have open-sourced all the code, data, and models in the following anonymized GitHub link: https://github.com/oneal2000/PRAG

• The paper introduces a novel Parametric RAG framework that embeds document knowledge into LLM parameters to minimize the burdens of in-context retrieval.
• It leverages low-rank adaptation techniques to convert external documents into compact parameter forms that integrate directly into feed-forward network layers.
• Experimental results show significant improvements over traditional RAG methods on multi-hop reasoning benchmarks and scalable performance across various LLM configurations.

The paper "Parametric Retrieval Augmented Generation" explores advancing Retrieval-Augmented Generation (RAG) with a paradigm shift from the conventional in-context knowledge injection to a parametric approach, herein termed Parametric RAG. Traditional RAG methods append retrieved documents to the input context of LLMs, effectively integrating external knowledge but incurring increased computational overhead and potentially degrading complex reasoning performances due to the expansion of input context length.

Key Concepts and Methodology:

1. Limitations of In-context RAG:
   • Computational Overhead: The inclusion of multiple documents inflates the input prompt, augmenting both processing time and the memory footprint.
   • Underutilization of Parametric Space: LLMs inherently store knowledge within their parameters rather than just the input context. This in-context method fails to capitalize on this storage, potentially limiting generation efficacy.
2. Introduction of Parametric RAG:
   • Parametric RAG proposes parameterizing external documents and integrating these parameters directly into an LLM's feed-forward network (FFN) layers. This integration effectively diminishes online computational costs and enhances the depth of knowledge integration.
   • Document Parameterization: Instead of varying the input context dynamically, documents are converted into a compact parametric form via low-rank matrix adaptations, thereby affecting the model's FFN during inference.
   • Retrieve-Update-Generate Workflow: A functional decomposition wherein:
     • Retrieve: Selecting top-n relevant documents based on a query.
     • Update: Merging and integrating parameterized document representations into the LLM.
     • Generate: Utilizing this updated model to produce contextually informed and accurate responses.
3. Parameterization Methodology:
   • Offline Document Augmentation: This involves document rewriting and the creation of QA pairs to enrich each document semantically before parameterization.
   • LoRA (Low-Rank Adaptation): Parameters are represented by updating the FFN matrices with low-rank increment matrices, facilitating easy and efficient document knowledge incorporation.
4. Experimental Validation:
   • The approach significantly outperforms traditional RAG baselines, demonstrating enhanced performance across multi-hop reasoning benchmarks like 2WikiMultihopQA and HotpotQA.
   • Performance is validated on multiple LLM configurations (e.g., LLaMA-1B, Qwen-1.5B), with findings indicating scalable improvements proportional to model size.
   • An exploratory integration of both parametric and in-context document representations notably maximized performance, suggesting potential applicability across diverse RAG scenarios.
5. Comparison with Existing Methods:
   • The study highlights the shortcomings of in-context methods, particularly in long-context processing inefficiencies, and the increased burden on computational resources.
   • Parametric representation shows a potential reduction in the need for extensive context windows, potentially alleviating attention bottlenecks in large models.

Conclusions and Future Directions:

The Parametric RAG framework introduces a novel method for knowledge integration into LLMs, directly modifying model parameters and allowing for the dynamic and efficient use of external knowledge sources. While this approach demonstrates promising improvements in managing computational overhead and scaling with large LLMs, challenges persist in optimizing the offline computational expense and generalizing parameter representations across models. Future research could explore more lightweight parametric encodings and improve the universality of document representations to enhance interoperability across varying LLM architectures. Additionally, exploring extensions into task-specific adjustments or further combination with traditional RAG methods presents a fertile ground for expanding the utility of parametric knowledge integration.