Beyond Sequential Reranking: Reranker-Guided Search Improves Reasoning Intensive Retrieval

Abstract: The widely used retrieve-and-rerank pipeline faces two critical limitations: they are constrained by the initial retrieval quality of the top-k documents, and the growing computational demands of LLM-based rerankers restrict the number of documents that can be effectively processed. We introduce Reranker-Guided-Search (RGS), a novel approach that bypasses these limitations by directly retrieving documents according to reranker preferences rather than following the traditional sequential reranking method. Our method uses a greedy search on proximity graphs generated by approximate nearest neighbor algorithms, strategically prioritizing promising documents for reranking based on document similarity. Experimental results demonstrate substantial performance improvements across multiple benchmarks: 3.5 points on BRIGHT, 2.9 on FollowIR, and 5.1 on M-BEIR, all within a constrained reranker budget of 100 documents. Our analysis suggests that, given a fixed pair of embedding and reranker models, strategically selecting documents to rerank can significantly improve retrieval accuracy under limited reranker budget.

• The paper introduces Reranker-Guided Search, a method that integrates reranker feedback into the retrieval process to enhance performance on reasoning-intensive tasks.
• Its methodology employs greedy search over proximity graphs using ANNS, achieving significant NDCG@10 improvements on BRIGHT, FollowIR, and M-BEIR benchmarks.
• The approach shows robustness to query embedding perturbations and demonstrates computational efficiency with limited reranker budgets.

Reranker-Guided Search for Improved Retrieval

Introduction

The paper "Beyond Sequential Reranking: Reranker-Guided Search Improves Reasoning Intensive Retrieval" (2509.07163) addresses limitations in traditional retrieve-and-rerank pipelines, particularly for reasoning-intensive tasks. Existing pipelines are often restricted by the quality of initial retrieval results and the computational demands of rerankers, particularly those based on LLMs. This paper introduces Reranker-Guided-Search (RGS), a novel approach that directly integrates reranker preferences into the document retrieval process, thereby enhancing retrieval accuracy within a constrained reranker budget.

Methodology

RGS leverages a greedy search over proximity graphs built using Approximate Nearest Neighbor Search (ANNS) algorithms. The central premise is to prioritize documents for reranking based on their similarity to previously favored documents, thereby exploiting the clustering hypothesis which states that similar documents are likely to be similarly relevant to a query. The approach consists of two stages:

1. Initial Retrieval: A list of seed documents is retrieved based on proximity to the query in the embedding space.
2. Iterative Refinement: Documents from the seed list are reranked, and new candidate documents are added based on their proximity to high-ranking documents. This process continues iteratively until the reranker budget is exhausted.

   Figure 1: An example from TheoremQA-T illustrating how Reranker-Guided-Search works, summarizing the query and document titles.

Experimental Results

The efficacy of RGS was demonstrated across three benchmarks: BRIGHT, FollowIR, and M-BEIR. The benchmarks cover a range of complex retrieval tasks including scientific reasoning and multimodal retrieval. RGS showed substantial improvements over traditional methods in these contexts:

• BRIGHT Benchmark: Improved NDCG@10 from 25.3 to 28.8 with a 100-document budget.
• FollowIR Benchmark: Achieved consistent gains with NDCG@10 scores reaching 63.3 compared to 60.4 using standard reranking.
• M-BEIR Benchmark: RGS improved retrieval accuracy from 25.9 to 31.0 with a limited reranker budget.

Figure 2: Average NDCG@10 result over a selected subset of datasets from BRIGHT, comparing RGS with other methods.

Analysis and Implications

RGS's performance was less sensitive to the choice of embedding models, as the method primarily relied on the reranker's ability to discern relevancy through similarity structures among documents. The study also revealed that retrieval accuracy depended more on the alignment of the reranker model with ground truth. Furthermore, RGS demonstrated robustness against perturbations in query embeddings, indicating its suitability for tasks that require reasoning-based retrieval.

Figure 3: Retrieval accuracy for different retrieval methods under varying levels of embedding perturbation.

Conclusion

Reranker-Guided-Search represents a significant advancement in retrieval methods, particularly for tasks that require intensive reasoning or multimodal analysis. By incorporating document similarity more directly into the retrieval process, RGS offers a more computationally efficient and accurate alternative to traditional sequential reranking pipelines. Future work may address potential challenges in parallelizing RGS to further enhance its applicability and efficiency in real-world retrieval systems.

Future Outlook

With the increasing complexity and scale of retrieval tasks, particularly those involving reasoning and multimodal queries, RGS provides a promising framework. Future developments might explore optimizing the search process further or integrating more sophisticated rerankers to enhance performance in even broader contexts. Additionally, examining the scalability of this approach across different domains could broaden its utility in diverse fields such as information retrieval and natural language processing.

In summary, Reranker-Guided-Search offers a valuable contribution to retrieval methodologies by effectively balancing retrieval accuracy and computational feasibility, opening pathways for more sophisticated, reasoning-intensive retrieval systems.