Towards Better Generalization in Open-Domain Question Answering by Mitigating Context Memorization

Abstract: Open-domain Question Answering (OpenQA) aims at answering factual questions with an external large-scale knowledge corpus. However, real-world knowledge is not static; it updates and evolves continually. Such a dynamic characteristic of knowledge poses a vital challenge for these models, as the trained models need to constantly adapt to the latest information to make sure that the answers remain accurate. In addition, it is still unclear how well an OpenQA model can transfer to completely new knowledge domains. In this paper, we investigate the generalization performance of a retrieval-augmented QA model in two specific scenarios: 1) adapting to updated versions of the same knowledge corpus; 2) switching to completely different knowledge domains. We observe that the generalization challenges of OpenQA models stem from the reader's over-reliance on memorizing the knowledge from the external corpus, which hinders the model from generalizing to a new knowledge corpus. We introduce Corpus-Invariant Tuning (CIT), a simple but effective training strategy, to mitigate the knowledge over-memorization by controlling the likelihood of retrieved contexts during training. Extensive experimental results on multiple OpenQA benchmarks show that CIT achieves significantly better generalizability without compromising the model's performance in its original corpus and domain.

• The paper introduces Corpus-Invariant Tuning (CIT) to reduce memorization and boost model generalizability.
• The study reveals that CIT-trained models excel in adapting to updated knowledge corpora and new domains.
• Experimental findings show significant improvements in exact match scores, highlighting CIT’s practical impact on QA systems.

Mitigating Context Memorization for Enhanced Generalizability in Open-Domain QA Models

Introduction

In the domain of Open-Domain Question Answering (OpenQA), incorporating a current and extensive knowledge corpus is crucial for accurate responses. Yet, knowledge evolves, requiring models to adapt swiftly to new information or entirely different domains. This study explores the generalization capabilities of retrieval-augmented QA models, pinpointing challenges related to the reader's overdependence on memorizing external corpus content. A novel training strategy, Corpus-Invariant Tuning (CIT), is proposed to reduce such over-memorization, thereby boosting generalizability without sacrificing performance on the original corpus.

Evaluation of Model Generalization

The research dissects generalization performance across two scenarios: adapting to updated knowledge corpus versions and transitioning to disparate knowledge domains. Using state-of-the-art retrieval-augmented models, the study reveals a notable performance dip when models are directly applied to updated corpora or different domains, even after additional tuning. This degradation is attributed to the reader component's tendency to memorize retrieved corpus content, which hampers adaptability to new or updated information.

Corpus-Invariant Tuning (CIT)

Responding to these challenges, the study introduces CIT, aimed at minimizing the reader's knowledge memorization during training. By regulating the reader's likelihood of retrieved documents and incorporating an innovative loss term, CIT discourages the embedding of retrieved knowledge into the reader's parameters. Extensive experiments demonstrate CIT's efficacy in improving the generalizability of OpenQA models across different corpora versions and unrelated domains, validated by significant improvements in exact match scores.

Experiments and Findings

Experiments across various benchmarks and settings reveal that models trained with CIT exhibit superior generalization without compromising their inherent performance. In scenarios involving updates to the knowledge corpus or shifts to new domains, CIT-trained models significantly outperform their counterparts. Furthermore, CIT's influence extends to the retrieval performance, enhancing the relevance of retrieved documents and the reader's reliance on them for answering questions.

Implications and Future Directions

This research underscores the criticality of addressing knowledge over-memorization in retrieval-augmented QA models to improve their adaptability and utility in real-world applications. CIT represents a pivotal step towards developing OpenQA systems capable of seamlessly adapting to evolving knowledge landscapes. Future work could explore automatic adjustment mechanisms for CIT's hyperparameters, tailoring the balance between knowledge retrieval and memorization to the specific requirements of diverse application contexts.

Conclusion

The study presents a compelling narrative on bolstering the generalization capabilities of OpenQA models through the mitigation of context memorization. By pioneering the Corpus-Invariant Tuning strategy, it sets a new paradigm for designing and training more adaptable, robust, and efficient question-answering systems.