ExcluIR: Exclusionary Neural Information Retrieval

Abstract: Exclusion is an important and universal linguistic skill that humans use to express what they do not want. However, in information retrieval community, there is little research on exclusionary retrieval, where users express what they do not want in their queries. In this work, we investigate the scenario of exclusionary retrieval in document retrieval for the first time. We present ExcluIR, a set of resources for exclusionary retrieval, consisting of an evaluation benchmark and a training set for helping retrieval models to comprehend exclusionary queries. The evaluation benchmark includes 3,452 high-quality exclusionary queries, each of which has been manually annotated. The training set contains 70,293 exclusionary queries, each paired with a positive document and a negative document. We conduct detailed experiments and analyses, obtaining three main observations: (1) Existing retrieval models with different architectures struggle to effectively comprehend exclusionary queries; (2) Although integrating our training data can improve the performance of retrieval models on exclusionary retrieval, there still exists a gap compared to human performance; (3) Generative retrieval models have a natural advantage in handling exclusionary queries. To facilitate future research on exclusionary retrieval, we share the benchmark and evaluation scripts on \url{https://github.com/zwh-sdu/ExcluIR}.

• The paper introduces exclusionary retrieval through the ExcluIR dataset and benchmark, providing a framework to assess document filtering capabilities.
• It examines limitations of sparse and dense retrieval models on exclusionary queries while noting the advantages of generative models.
• Rigorous experiments using metrics like Recall@N and MRR highlight a performance gap that prompts further refinement in IR systems.

Exploration of Exclusionary Retrieval in Document Search: Introducing ExcluIR Benchmark and Dataset

Overview

This paper introduces the novel concept of exclusionary retrieval in document search, a scenario wherein users specifically indicate content they wish to exclude from search results. To facilitate research in this area, the authors developed ExcluIR, which includes both an evaluation benchmark and a training dataset, filled with exclusionary queries to test and train retrieval models' abilities to understand and process such queries effectively.

Key Contributions

1. ExcluIR Dataset: The dataset comprises 70,293 exclusionary queries each paired with a positive and a negative document, aiming to study whether models can discern documents to exclude based on user queries.
2. Benchmark Creation: A subset of the dataset, consisting of 3,452 human-annotated exclusionary queries, forms the benchmark for evaluating the ability of information retrieval systems to handle exclusionary queries.
3. Comprehensive Analysis: An in-depth examination of existing retrieval models like sparse, dense, and generative reveals their limitations and capabilities concerning exclusionary retrieval tasks.

Observational Insights

• Struggles of Current Models: Existing retrieval architectures demonstrate a clear challenge in effectively understanding and processing exclusionary queries.
• Generative Model Advantages: Generative retrieval models inherently exhibit better performance due to their ability to contextually generate answers, which seems advantageous for handling the nuances of exclusionary queries.
• Room for Improvement: Even with targeted training data, there remains a significant performance gap relative to human benchmarking, indicating substantial room for model improvement or perhaps a need for new approaches in model architecture.

Dataset and Methodology

The construction of ExcluIR followed meticulous steps to ensure quality:

1. Query Generation: Utilized ChatGPT to generate exclusionary queries from document pairs sourced from HotpotQA. This included refining queries for relevance and complexity.
2. Manual Corrections: Employed human reviewers to ensure the naturalness and accuracy of generated queries, making necessary modifications to maintain data quality.
3. Quality Control: Implemented rigorous checks and balances via worker feedback and random checks to maintain the high standard of the dataset.

Experimental Setup

• The research evaluated several models, categorizing them into sparse, dense, and generative retrieval types.
• Key metrics used include Recall@N, MRR (Mean Reciprocal Rank), and special metrics designed to evaluate exclusionary retrieval, highlighting differences in the ranking of positive and negative documents as influenced by exclusionary queries.

Discussion and Future Work

The findings suggest that while there is some progress in handling exclusionary queries by leveraging specifically designed training sets, the overall effectiveness is still not at par with human levels, especially in sophisticated real-world scenarios. Future research could explore multi-round exclusionary contexts or develop more nuanced generative models that can better understand and generate context-aware responses to exclusionary prompts.

Conclusion

This study paves the way for further discussions and developments in the field of exclusionary retrieval. The ExcluIR benchmark and datasets are a significant step forward, providing the necessary tools and foundational work to spur future enhancements and innovations in document retrieval systems.