Retriever Portfolios: A Principled Approach to Adaptive RAG

Abstract: Retrieval-augmented generation (RAG) systems typically rely on a single retriever and a single set of hyperparameters, despite facing highly heterogeneous queries that range from simple factoid questions to complex multi-hop reasoning. We propose a method that automatically selects a small, diverse subset of retrievers (a portfolio) from a large pool of candidates, to cover different regions of the target query distribution. We formalize this setting via an expected best-of-$k$ objective over the query distribution and show that it admits an efficient portfolio construction algorithm with near-optimal guarantees. Across multiple QA benchmarks, our learned portfolios and router pipeline consistently outperform single-retriever and naive multi-retriever baselines on both retrieval metrics and answer quality. In addition, compared to inference-time hyperparameter tuning approaches, fixed portfolios enable parallel retrieval and LLM calls, achieving comparable (and sometimes better) accuracy with substantially lower latency and token cost.

• The paper demonstrates that a portfolio of diverse retrievers can be selected using submodular optimization to maximize per-query performance and retrieval coverage.
• It presents a greedy algorithm with theoretical approximation guarantees, ensuring efficient portfolio construction with minimal query samples.
• Empirical evaluations on QA benchmarks show improved accuracy, reduced latency, and lower token usage compared to fixed or adaptive single retrievers.

Retriever Portfolios: Adaptive RAG via Principled Submodular Optimization

Motivation and Problem Setting

Retrieval-augmented generation (RAG) architectures, which ground LLMs via retrieval from external corpora, typically deploy a single retriever and a fixed hyperparameter configuration for all queries. This practice is fundamentally suboptimal due to inherent heterogeneity in user queries: some are simple factoid lookups, others require complex or multi-hop reasoning, and yet others require domain- or terminology-specific retrieval. Empirical and theoretical evidence demonstrates that distinct retrieval configurations excel on distinct subpopulations of the query distribution, and that no single retriever or hyperparameterization suffices for maximized retrieval coverage or downstream answer accuracy.

The central objective of the paper "Retriever Portfolios: A Principled Approach to Adaptive RAG" (2605.31176) is to formalize and solve the following problem: given a large heterogeneous pool of candidate retrievers, how can one select a small, diverse portfolio of $k$ retrievers such that for any random query from the target distribution, at least one portfolio member is highly performant? The aim is to simultaneously maximize retrieval coverage, enable practical per-query adaptation, and maintain tight computational budgets and inference-time latency.

Formalization: Expected Best-of-$k$ Objective and Submodular Portfolio Selection

Let $\mathcal{Q}$ denote the query distribution, $\mathcal{R}=\{r_1,\ldots,r_m\}$ the retriever pool, and $s(q,r):\mathcal{Q}\times\mathcal{R}\to[0,1]$ a black-box scoring function (such as retrieval recall or answer accuracy). Instead of maximizing average retriever performance, the key criterion is:

$$F(S) = \mathbb{E}_{q\sim \mathcal{D}} [\max_{r\in S} s(q,r) ]$$

where $S\subseteq \mathcal{R}$, $|S| \leq k$.

Crucially, $F(S)$ encourages diversity and coverage—by focusing on the maximum per-query score, it incentivizes portfolios whose members specialize in complementary subregions of the query space, so that rare or out-of-distribution queries are not neglected.

The portfolio optimization reduces to maximizing a monotone submodular function under a cardinality constraint. The paper leverages this structure to show that a straightforward sample-based greedy algorithm yields a $(1-1/e - \epsilon)$ approximation guarantee to the optimal portfolio, requiring only $k$0 query samples—provably tractable even for large retriever pools.

Retriever Portfolio Construction Algorithm

The greedy selection procedure iteratively adds the retriever with the largest marginal gain over the empirical objective (computed by sampling $k$1 queries and caching $k$2 matrices). This approach amortizes all adaptation costs offline. The resulting portfolio comprises a small number of retrievers (typically $k$3), each with distinct architectural or hyperparameter characteristics, covering the heterogeneous query space far better than any single retriever or top-$k$4 average-performer ensemble.

Architecture: Adaptive Routing and Parallelized RAG Pipeline

At inference time, instead of executing the entire portfolio exhaustively, the pipeline employs a lightweight learned router (a neural encoder aggregating raw query text and backbone-specific embeddings) to dynamically rank the $k$5 portfolio members given an incoming query. For each query, the router selects the top $k$6 (e.g., $k$7 or $k$8) retrievers, executes retrieval and answer generation in parallel, and aggregates answers via an LLM-based selector. This architecture maintains low latency, reduces total token and compute consumption, and scales efficiently to demanding QA settings.

(Figure 1)

Figure 1: Adaptive inference pipeline with query encoding, portfolio routing, parallel retrieval/generation, and selector-based answer aggregation.

Empirical Evaluation: QA Benchmarks, Coverage, and Efficiency

The evaluation spans four open-domain and multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA, TriviaQA, and MusiQue) with supporting document labels, using two open-weight answer LLMs (Gemma-3-27B-It and Llama-3.1-70B-Instruct) and extensive candidate retriever pools (360 variants including diverse dense, diversified, diversity-relevance-tradeoff, and graph-based types). Empirically:

• Retrieval Coverage: Portfolios selected by the best-of-$k$9 (as opposed to mean retriever score) dramatically improve support recall and F1 compared to both the best single retriever and the $\mathcal{Q}$0 best retrievers by average score. The coverage gains cannot be attributed merely to increasing the number of documents: retrieving many more documents with one retriever yields higher recall but much lower precision (F1), whereas portfolios achieve both high recall and F1 by admitting only complementary, specialized retrievers.

(Figure 2)

Figure 2: Support recall improves with portfolio size $\mathcal{Q}$1; learned portfolios (blue) vastly outperform average-best baselines and retrieve-more-document controls.

• Downstream QA Accuracy: Routing queries over portfolio members using the learned router yields substantial gains in exact match (EM) answer accuracy across all datasets and models, outperforming adaptive approaches like Vendi-RAG (which perform expensive per-query retrieval-parameter tuning at inference) as well as mixture-of-retriever or best-single-retriever controls. The improvement persists as model size increases and across diverse question types.

Figure 3: Exact match (EM) as a function of retrieval strategy for Gemma-3-27B-It on multiple QA datasets; all-pool portfolios dominate single retriever and adaptive baselines.

• Cost/Latency and Efficiency: Fixed portfolios, in combination with the router and parallel answer execution, substantially reduce wall-clock latency and total generated tokens compared to inference-time adaptive approaches such as Vendi-RAG, while matching or surpassing their accuracy. This efficiency gain is attributed to amortizing adaptation offline and leveraging parallel inference.

Figure 4: Tokens generated versus EM score for Gemma27B: offline-optimized portfolios provide a sharply better efficiency-accuracy trade-off compared to inference-time tuning baselines.

Theoretical and Practical Implications

The work rigorously connects RAG system design to the literature on algorithm portfolios, submodular coverage, and data-driven combinatorial optimization. It provides strong approximation guarantees for the construction algorithm, minimal sample complexity and memory requirements, and empirical validation that the structural properties of the best-of-$\mathcal{Q}$2 objective yield real-world improvements.

Practically, these insights enable RAG system designers to:

• Customize retrieval adaptively with bounded compute/latency
• Handle distribution shifts and hard or rare queries without requiring dynamic online search or manual intervention
• Integrate new retriever innovations or domain-specific methods into the portfolio as the field advances

Future Extensions

Potential future avenues include further scaling and automation of the retriever pool (integrating more architectures, multi-modal retrievers, or plug-in retrievers for dedicated domains), extending portfolio selection to accommodate non-stationary or evolving query distributions (via continual or periodic portfolio re-optimization), and exploring theoretical guarantees under bandit or lifelong learning regimes.

Conclusion

This paper establishes a rigorous, sample-efficient, and practically effective approach for adaptive RAG via retriever portfolios. By casting retriever selection as a submodular maximization over expected query coverage, and providing algorithmic solutions with tight guarantees, the work advances both theoretical and empirical foundations of retrieval-augmented LLMs in heterogeneous, real-world deployments. The portfolio approach offers a generalizable blueprint for query-adaptive information retrieval, promising tangible improvements in downstream accuracy, robustness, and deployment efficiency across large-scale, diverse QA settings.