ECLIPSE: Contrastive Dimension Importance Estimation with Pseudo-Irrelevance Feedback for Dense Retrieval

Abstract: Recent advances in Information Retrieval have leveraged high-dimensional embedding spaces to improve the retrieval of relevant documents. Moreover, the Manifold Clustering Hypothesis suggests that despite these high-dimensional representations, documents relevant to a query reside on a lower-dimensional, query-dependent manifold. While this hypothesis has inspired new retrieval methods, existing approaches still face challenges in effectively separating non-relevant information from relevant signals. We propose a novel methodology that addresses these limitations by leveraging information from both relevant and non-relevant documents. Our method, ECLIPSE, computes a centroid based on irrelevant documents as a reference to estimate noisy dimensions present in relevant ones, enhancing retrieval performance. Extensive experiments on three in-domain and one out-of-domain benchmarks demonstrate an average improvement of up to 19.50% (resp. 22.35%) in mAP(AP) and 11.42% (resp. 13.10%) in nDCG@10 w.r.t. the DIME-based baseline (resp. the baseline using all dimensions). Our results pave the way for more robust, pseudo-irrelevance-based retrieval systems in future IR research.

• The paper presents Eclipse, a method that leverages pseudo-irrelevance feedback to contrastively estimate dimension importance and enhance dense retrieval.
• The methodology uses a centroid of irrelevant documents to filter noise, achieving improvements of up to 19.50% in mean AP and 11.42% in nDCG@10.
• The findings highlight that contrastive learning in retrieval systems can effectively balance dimensionality reduction with enhanced search accuracy.

Eclipse: A Novel Approach in Dense Information Retrieval Through Contrastive Dimension Importance Estimation

The paper, titled "Eclipse: Contrastive Dimension Importance Estimation with Pseudo-Irrelevance Feedback for Dense Retrieval," presents an innovative methodology to enhance dense retrieval systems by defining an approach referred to as Eclipse. The authors articulate the limitations of existing dense retrieval models which, although proficient in embedding text into high-dimensional spaces, struggle with the effective separation of relevant signals from non-relevant noise. This research aims to address these shortcomings by leveraging information from both relevant and non-relevant documents.

The premise of the study is grounded in the Manifold Clustering Hypothesis, which posits that relevant documents lie on a lower-dimensional manifold specific to each query. While the hypothesis has informed the development of new retrieval strategies, existing methods still inaccurately discriminate between pertinent and extraneous information. In this context, Eclipse introduces a novel conceptualization by utilizing both relevant and irrelevant documents to estimate the importance of dimensions, enhancing retrieval performance significantly.

Methods and Results

The proposed Eclipse method employs a centroid based on irrelevant documents as a reference to estimate noisiness in relevant ones. This framework is distinct from traditional approaches that predominantly consider relevant feedback, potentially retaining noisy dimensions that can compromise retrieval efficacy. The methodological innovation of Eclipse is its use of pseudo-irrelevance feedback, offering a contrastive learning approach that distinguishes between significant and insignificant dimensions in document embeddings.

The performance of this new method was rigorously tested against three dense retrieval models: ANCE, Contriever, and TAS-B, using multiple benchmark datasets such as TREC Deep Learning 2019, 2020, Deep Learning Hard 2021, and Robust 2004. The empirical results are compelling; Eclipse demonstrates an average enhancement of up to 19.50% in mean Average Precision (AP) and 11.42% in normalized Discounted Cumulative Gain at rank 10 (nDCG@10) relative to the DIME-based baseline. Even more striking is a comparative improvement of up to 22.35% in AP and 13.10% in nDCG@10 against the baseline model using all dimensions.

These findings are underscored by Eclipse's robust numerical outcomes across different configurations of document embeddings, indicating that the method not only refines the retrieval process through better dimensionality reduction but also achieves efficiency in resource utilization by maintaining high performance even at 50% of original dimensionality.

Implications and Future Directions

The introduction of Eclipse has significant implications for the field of Information Retrieval (IR) both practically and theoretically. By demonstrating that contrastive learning with pseudo-irrelevance feedback can notably enhance retrieval effectiveness, the study broadens the scope of existing retrieval models that typically emphasize only relevant documents. The dual approach of examining both relevant and non-relevant dimensions offers a promising new direction for refining search efficiency and reliability in dense IR.

Looking ahead, the potential applications of Eclipse could extend to other domains within natural language processing, such as question answering and document summarization, where understanding and leveraging relevant dimensionality is crucial. Furthermore, the integration of such methodologies with LLMs could lead to the development of even more capable and nuanced AI systems.

The study encourages future research to explore additional strategies for identifying the most relevant dimensions, particularly those that can effectively elevate topically significant documents in ranking systems. The pursuit of advanced methods to fine-tune the balance between dimensionality and retrieval performance could yield further improvements in the efficacy of dense information retrieval systems.