Principled and Scalable Diversity-Aware Retrieval via Cardinality-Constrained Binary Quadratic Programming

Abstract: Diversity-aware retrieval is essential for Retrieval-Augmented Generation (RAG), yet existing methods lack theoretical guarantees and face scalability issues as the number of retrieved passages $k$ increases. We propose a principled formulation of diversity retrieval as a cardinality-constrained binary quadratic programming (CCBQP), which explicitly balances relevance and semantic diversity through an interpretable trade-off parameter. Inspired by recent advances in combinatorial optimization, we develop a non-convex tight continuous relaxation and a Frank--Wolfe based algorithm with landscape analysis and convergence guarantees. Extensive experiments demonstrate that our method consistently dominates baselines on the relevance-diversity Pareto frontier, while achieving significant speedup.

• The paper introduces a principled CCBQP framework that optimizes passage relevance and semantic diversity using an interpretable trade-off parameter.
• It leverages a tight non-convex relaxation and a specialized Frank–Wolfe algorithm to achieve scalable, efficient retrieval with exact convergence.
• Empirical evaluations show superior recall-diversity trade-offs and up to 22.9× speedups over traditional methods in large-scale retrieval scenarios.

Principled and Scalable Diversity-Aware Retrieval via CCBQP: An Expert Analysis

Motivation and Problem Statement

Retrieval-Augmented Generation (RAG) fundamentally relies on the quality of retrieved passages to ground LLMs in external knowledge, eliminating hallucinations and boosting factuality. However, standard top-$k$ retrieval frequently results in redundant, semantically overlapping passages due to corpus-level duplications, constraining the effective utilization of LLM context and undermining coverage across multifaceted queries. Existing diversity-aware retrieval techniques, notably Maximal Marginal Relevance (MMR) and Determinantal Point Processes (DPP), struggle with theoretical guarantees and scalability: MMR lacks formal approximation guarantees for non-monotone objectives and scales linearly in $k$, while DPP suffers from both computational intractability (NP-hard MAP inference) and uninterpretability of the relevance-diversity trade-off parameter.

This paper introduces a rigorous formulation for diversity-aware retrieval as a cardinality-constrained binary quadratic programming (CCBQP), explicitly optimizing relevance and semantic diversity with an interpretable trade-off parameter, and overcomes both scalability and theoretical limitations through a tight non-convex relaxation solved efficiently via a specialized Frank--Wolfe algorithm.

Formal Framework and Theoretical Guarantees

The proposed formalization selects $k$ passages maximizing a combined objective: passage relevance to the query and semantic diversity, where diversity is quantified by negative pairwise cosine similarity in embedding space. The crucial trade-off is controlled by parameter $\theta$, with proper scaling for both terms, ensuring $\theta$ meaningfully interpolates between relevance and diversity across different $k$.

The authors leverage a non-convex continuous relaxation of the CCBQP, showing tightness of relaxation (integral solutions coincide with the binary optimum for $\lambda \geq 2$) under realistic embedding assumptions. Gradient-based optimization benefits from favorable landscape properties: all stationary points are either integral maximizers or strict saddles with positive curvature, enabling algorithms to escape suboptimal regions efficiently.

Moreover, as $\lambda$ increases, the space of local maximizers monotonically expands, potentially introducing spurious optima but preserving original maximizers—thus, the authors adopt $\lambda=2$ as a balance.

Optimization Algorithmic Innovations

The tailored Frank--Wolfe algorithm incorporates several key enhancements:

• Exact Line Search: By leveraging the quadratic structure, the optimal step size for each iteration can be solved in closed form ($\gamma$ maximizing a parabola), bypassing the convergence bottlenecks due to high Lipschitz constants caused by clustered embeddings.
• Sparse GEMV Acceleration: Computations involving matrix-vector products are reduced via sparsity exploitation, dropping complexity from $k$0 to $k$1. This is crucial for large candidate pools and enables sublinear scaling in $k$2.
• Local Exact Convergence: The algorithm rapidly converges to discrete maximizers without the need for rounding, a direct consequence of landscape guarantee.
• Complexity Analysis: Per-iteration complexity is $k$3 but exhibits practical sublinear scaling in $k$4, significantly outperforming extant greedy methods.

Empirical Evaluation: Retrieval and Generation

Retrieval Quality and Efficiency

Experiments on ASQA and QAMPARI with candidate pool sizes $k$5 million evaluate Recall@$k$6 and Intra-List Average Distance (ILAD) as metrics for relevance and diversity, respectively. The trade-off parameter $k$7 is systematically varied. The method consistently dominates baselines (MMR, DPP) on the relevance-diversity Pareto frontier, avoiding local sacrifices of recall typical in greedy methods and achieving significant speedups—up to $k$8 faster than MMR for large $k$9.

At moderate $k$0 (practically relevant for RAG), the proposed method achieves superior recall-diversity balance at reduced latency. For high $k$1, all methods revert to standard top-$k$2 retrieval, but the Frank--Wolfe algorithm remains much more efficient.

Figure 1: Pareto frontier and computational latency for ASQA ($k$3), highlighting the superior relevance-diversity trade-off and sublinear scaling in $k$4.

Generation Quality

End-to-end generation experiments emphasize the downstream performance on QAMPARI, using Llama-3.1-8B-Instruct as the reader LLM. Passage diversity, when promoted judiciously ($k$5–$k$6), yields modest but consistent improvements in F1 over standard top-$k$7 retrieval. Excessive diversity ($k$8) can degrade performance due to displacement of relevant evidence, especially in corpora with inherently low redundancy (e.g., Wikipedia).

Increasing retrieval size $k$9 increases recall but reduces precision, suggesting noise-induced information overload for the reader LLM. Generation results for ASQA show comparable trends: diversity-aware retrieval robustly improves recall at practical $\theta$0, while retaining algorithmic scalability and hyperparameter robustness.

Practical and Theoretical Implications

The introduced CCBQP framework provides a principled foundation for diversity-aware retrieval, supporting interpretable and rigorous control of relevance-diversity balance, and establishing an efficient optimization regime in real-world RAG settings. The scalability makes it highly amenable to extended context windows in modern LLMs, enabling retrieval of large, diverse passage sets without inferential bottlenecks.

From a theoretical standpoint, the tight relaxation and benign optimization landscape advance the tractability for NP-hard subset selection problems involving relevance-diversity trade-offs, and demonstrate that non-convex methods can yield exact solutions with appropriately structured objectives.

Future research directions include:

• Evaluation on High-Redundancy Corpora: To fully exploit diversity-aware retrieval, more redundant datasets must be assessed.
• Larger Model Deployment: Extending experiments to larger LLMs may uncover novel behaviors and further validate scalability.
• Dynamic Trade-off Optimization: Joint optimization of $\theta$1 in context-dependent settings, as supported by recent theoretical results.
• Integration with Advanced RAG Pipelines: Combining the framework with rerankers or hierarchical evidence selection modules.

Conclusion

This work establishes a rigorous, scalable, and interpretable framework for diversity-aware retrieval under the RAG paradigm, addressing key theoretical and computational deficiencies of prior methods. The tight continuous relaxation, specialized Frank--Wolfe algorithm, and extensive empirical validation demonstrate both robust Pareto frontier quality and exceptional efficiency. These results position CCBQP-based retrieval as a foundational tool for advanced knowledge-intensive NLP systems, facilitating context expansion and diversity-controlled generation in next-generation LLM architectures.