Parallel Context-of-Experts Decoding for Retrieval Augmented Generation

Abstract: Retrieval Augmented Generation faces a trade-off: concatenating documents in a long prompt enables multi-document reasoning but creates prefill bottlenecks, while encoding document KV caches separately offers speed but breaks cross-document interaction. We propose Parallel Context-of-Experts Decoding (Pced), a training-free framework that shifts evidence aggregation from the attention mechanism to the decoding. Pced treats retrieved documents as isolated "experts", synchronizing their predictions via a novel retrieval-aware contrastive decoding rule that weighs expert logits against the model prior. This approach recovers cross-document reasoning capabilities without constructing a shared attention across documents.

• The paper introduces PCED, a training-free, decode-time framework that shifts evidence aggregation from cross-document attention to retrieval-weighted contrastive decoding.
• Empirical results indicate PCED outperforms parallel KV cache methods by up to +70 points in QA tasks and achieves a 180× speedup in time-to-first-token.
• The approach scales effectively with an increasing number of experts, robustly suppressing distractors and supporting high-throughput, multi-hop reasoning.

Parallel Context-of-Experts Decoding for Retrieval Augmented Generation: Technical Analysis

Motivation and Background

Retrieval Augmented Generation (RAG) enhances LLM factuality by conditioning generation on external corpora. While concatenating multiple documents into long input contexts enables multi-hop reasoning, it incurs substantial prefill latency and introduces attention bottlenecks. Furthermore, evidence distributed across concatenated documents can be ineffectually aggregated, leading to notable reasoning failures in long-context generation (2601.08670). Alternative methods, leveraging parallel KV cache encoding, improve inference efficiency but degrade performance in knowledge-intensive, multi-hop scenarios, as they eliminate cross-document attention required for evidence integration.

Framework: Parallel Context-of-Experts Decoding (Pced)

Pced is a training-free, decode-time expert aggregation framework. It moves evidence synthesis from cross-document attention to a contrastive decoding regime. Each retrieved document is paired with a precomputed KV cache, instantiating an independent expert. At each token step, Pced synchronizes the vote of all experts (retrieved documents and amateur/no-context prior) via a retrieval-weighted contrastive decoding rule:

• Per-expert logits $s_k$ are calibrated with the prior $s_0$ and gated by normalized, fused retrieval and reranker relevance scores $r_k$ through a contrastive objective parameterized by $\beta_0$ and $\gamma$.
• Token generation proceeds by selecting the maximum score over all expert candidates, allowing for dynamic, token-level switching between documents as generation progresses.

This modular cache-native design obviates the need for shared attention contexts, enabling scalable multi-document reasoning.

Figure 1: PCED framework: parallel expert decoding over independent document caches, with retrieval-gated evidence aggregation at each decoding step.

Empirical Results and Numerical Analysis

Pced exhibits strong empirical gains on benchmarks necessitating cross-document reasoning.

• On LOFT and LongBench, Pced outperforms parallel KV cache methods (e.g., APE, CacheBlend) by up to +70 absolute points in multi-document QA tasks and matches or exceeds full-context concatenation in 11/16 settings, despite encoding each context separately.
• MapReduce, which aggregates via per-document summarization and a second-pass synthesis, is outperformed by Pced by +23 points on challenging datasets (e.g., HotpotQA with Llama-3.1-8B: MapReduce 41 vs. PCED 64).
• In single-document tasks with irrelevant distractors, Pced Dense improves accuracy from 58 (top-1 context) and 79 (all) to 85 (NQ, Llama-3.1), demonstrating that retrieval-gated expert isolation effectively suppresses noise.

Ablation studies confirm both contrastive ($\beta_0$) and retrieval-gating ($\gamma$) contributions are necessary:

Analysis Condition	HotpotQA (Llama)	NQ (Llama)	HotpotQA (Mistral)	NQ (Mistral)
Only Contrastive ($\gamma=0$)	46	52	57	71
Only Retrieval ($\beta=0$)	53	70	65	80
Full PCED	64	85	66	81

Pced’s performance remains stable as the number of experts ($k$) increases from 8 to 128, indicating that its retrieval prior robustly suppresses distractor noise, enabling high recall without accuracy degradation.

Figure 3: PCED accuracy stability across top-$k$ expert pool sizes, with no significant loss up to $k=128$.

In terms of latency:

• Pced achieves an over 180× speedup in time-to-first-token compared to context concatenation for large candidate sets (TTFT: 0.14s vs 25.5s), as prefill computation is fully eliminated via stored expert caches.
• Total end-to-end latency is reduced by $\sim$1.7× on long input workloads.

  Figure 4: TTFT and end-to-end latency benchmarks: PCED scalably lowers latency across context sizes and candidate set cardinality.

Decoding Dynamics and Multi-hop Reasoning

PCED’s evidence aggregation is dynamic. It can hop between documents mid-generation depending on information needs, permitting robust multi-hop answering.

Figure 5: HotpotQA expert trace: PCED dynamically pivots between supporting documents at generation time.

Token-wise aggregation via the maximum scoring expert produces sharper expert switching than mixture or product-of-expert schemes, especially in multi-hop conditions. Mixture aggregation can slightly outperform in single-support cases, but max aggregation provides decisive gains for reasoning tasks (HotpotQA: Max 64 vs MoE 56).

Theoretical and Practical Implications

The PCED paradigm demonstrates that most gains from cross-document attention can be recovered by aggregating document evidence at decode time, without joint contextual encoding. This decouples prefill cost from reasoning ability, supporting high-throughput, scalable multi-document QA.

In practical systems, PCED is especially beneficial for read-heavy inference over static corpora such as enterprise KBs, where precomputed KV cache storage amortizes the cost of offline encoding. The method’s reliance on raw model logits restricts deployment to open/self-hosted models, but future work may extend the approach to distill the expert aggregation into models with restricted output interfaces.

Potential future directions include training models for native parallel context input and learned expert selection, potentially further optimizing both retrieval and aggregation phases. Additionally, hybrid schemes combining score-based gating with parametric model selectors may offer further accuracy and robustness improvements.

Conclusion

PCED shifts evidence synthesis in RAG from prefill attention to modular, retrieval-aware decode-time aggregation. It achieves substantial accuracy improvements over legacy parallel cache methods and context concatenation while drastically reducing latency, opening a pathway for scalable, high-recall, multi-document reasoning in LLMs. This reframing advances both the theoretical understanding of evidence integration in modular context architectures and their practical utility in retrieval-augmented systems.