Diverse Evidence, Better Forecasts: Multi-Agent Deliberation Under Information Asymmetry

Abstract: Multi-agent systems are increasingly used for forecasting future events, as deliberation among multiple LLMs is believed to improve reasoning and calibration. Yet existing approaches overlook a critical design choice: what information each agent receives. When all agents are given identical evidence, deliberation collapses into herding rather than genuine belief revision, leaving multi-agent systems little better than a single agent. We identify this as a fundamental gap and propose designed information asymmetry to close it: by partitioning evidence into shared public and disjoint private subsets, each agent holds exclusive knowledge that can only reach others through deliberation. We theoretically show that this decomposition reduces inter-agent error correlation, and instantiate it in InfoDelphi, a framework combining relevance-aware evidence routing, rationale-based iterative deliberation, and confidence-weighted aggregation. On PolyGym, a benchmark of 375 binary forecasting questions derived from real-world prediction markets, InfoDelphi outperforms the strongest single-agent and multi-agent baselines by 12--18% in Brier score and 4--8 percentage points in accuracy. More detailed experiments confirm that removing information asymmetry eliminates most deliberation gains, establishing diversity of input as the key enabler of effective multi-agent reasoning.

• The paper introduces designed evidence asymmetry, demonstrating that partitioning information into public and private subsets reduces inter-agent error correlation.
• It details the InfoDelphi framework with BM25-based evidence routing, multi-round rationale sharing, and confidence-weighted aggregation to enhance forecast accuracy.
• Empirical results on the PolyGym benchmark show significant improvements in Brier score and accuracy compared to homogeneous evidence setups.

Multi-Agent Deliberation with Designed Information Asymmetry for Probabilistic Forecasting

Introduction and Motivation

Forecasting future events using multi-agent LLM systems presents a critical challenge: traditional multi-agent deliberation frameworks typically assign all agents the same evidence pool, inadvertently inducing strong error correlation and diminishing ensemble gains. This work posits that diversity of evidence across agents is essential for productive collective reasoning. The paper introduces designed information asymmetry as a principle, partitioning the evidence into shared public and disjoint private subsets per agent. The InfoDelphi framework operationalizes this principle, highlighting the theoretical and empirical failures of conventional homogeneous-input debate (Figure 1).

Figure 1: Effect of information asymmetry on a real prediction—information-diverse deliberation enables self-correction, drastically reducing aggregate error.

Theoretical Foundations

The core theoretical innovation is the formal analysis of error correlation within agent ensembles as a function of shared versus private input. Under homogeneous evidence allocation, inter-agent error correlation approaches unity, nullifying ensemble calibration benefits. By contrast, the paper demonstrates that public/private evidence partitioning monotonically reduces this correlation.

Letting $\rho$ denote the fraction of shared evidence and assuming independent noise in private subsets, inter-agent error correlation is shown to be:

$$\mathrm{Corr}(\epsilon_i, \epsilon_j) = \frac{\rho^2\,\sigma_{\mathrm{pub}}^2}{\rho^2\,\sigma_{\mathrm{pub}}^2 + (1-\rho)^2\,\sigma_{\mathrm{priv}}^2}$$

A critical outcome is the existence of an optimal $\rho^* \in (0,1)$ that balances diversity and communicability. If $\rho=1$, agents perfectly correlate (homogeneous input); if $\rho=0$, agents are independent but lack common ground for meaningful deliberation. The framework mandates rationale sharing to permit transfer of private insights during deliberation rounds, formally breaking the Markov chain structure that otherwise prevents any increase in mutual information with the ground truth.

The InfoDelphi Framework

InfoDelphi operates in three phases: evidence routing, multi-agent deliberation with rationale sharing, and confidence-weighted forecast aggregation.

Figure 2: InfoDelphi framework overview—evidence is partitioned via BM25, agents deliberate via rationale exchange, and outputs are aggregated by confidence-weighted pooling.

Evidence Routing: Documents are ranked by BM25 relevance; top-$k$ form a public pool ($\rho \cdot |\mathcal{E}|$), and the remainder are shuffled round-robin into private agent-specific subsets to maximize private signal diversity. Each agent thus receives the public pool and a mutually exclusive set of supporting documents.

Multi-Agent Deliberation: Agents engage in $R$ rounds of exchange. Each round, an agent observes (i) their evidence subset and (ii) a summary of other agents' previous prediction rationales and scores. Rationale sharing is crucial for the propagation of private information, as only reasoning statements—not raw evidence—are visible cross-agent.

Confidence-Weighted Aggregation: Each agent outputs a probability. Final aggregation is performed in logit space, weighted by each agent's confidence (measured as $|\mathrm{logit}(p_j)|$), which amplifies contributors whose private evidence yields more extreme, and empirically more accurate, forecasts.

Experimental Results

PolyGym Benchmark

To isolate the effects of information asymmetry from retrieval variability, the PolyGym benchmark is introduced. It comprises 375 binary Polymarket questions, each coupled with up to 30 pre-retrieved, temporally filtered evidence snippets, enabling reproducible, controlled comparisons (Figure 3).

Figure 3: PolyGym Data Construction Pipeline ensures controlled, reproducible evaluation by providing identical, temporally filtered evidence to all methods.

Forecasting Performance

On PolyGym, InfoDelphi demonstrates robust empirical superiority:

• Brier score: 0.178 (InfoDelphi) vs. 0.202 (Standard Debate) and 0.206 (InfoDelphi without deliberation)
• Accuracy: 77.9% (InfoDelphi) vs. 73.9% (Standard Debate)

These results are 12–18% better in Brier score and 4–8 percentage points in accuracy over the strongest single-agent and traditional multi-agent baselines. The gains are directly attributed to information asymmetry and rationale sharing: naive ensembling, additional prompting, or increasing agent counts in homogeneous settings provides little to no improvement.

Ablations and Analysis

• Rationale sharing: Removing rationale text exchange destroys the deliberation benefit; accuracy drops to 70.4%.
• Evidence partitioning: Both insufficient ($\rho=0.3$) and excessive ($$\mathrm{Corr}(\epsilon_i, \epsilon_j) = \frac{\rho^2\,\sigma_{\mathrm{pub}}^2}{\rho^2\,\sigma_{\mathrm{pub}}^2 + (1-\rho)^2\,\sigma_{\mathrm{priv}}^2}$$0) public pools degrade performance, supporting the predicted optimality of a balanced split.
• Aggregation: Confidence-weighted pooling is critical; uniform aggregation reduces Brier score performance.
• Routing: BM25-based allocation outperforms random evidence assignment.

InfoDelphi's improvements generalize across GPT-5.4-mini, DeepSeek-V3.2, and Llama-4-Scout, with stronger underlying models leveraging rationale exchange more effectively.

Information Extremity and Diversity Dynamics

Agents assigned private evidence under InfoDelphi often produce more extreme (higher confidence) predictions. Empirical extremity correlates tightly with accuracy: the highest-confidence quartile achieves 94.7% accuracy and a Brier score of 0.052, verifying the theoretical basis for extremity-based weighting strategies (Figure 4).

Figure 4: Higher prediction extremity aligns with increased forecast accuracy, justifying confidence-weighted aggregation.

Diversity is preserved even after two rounds of rationale sharing—BM25-routed InfoDelphi only reduces inter-agent prediction variance by 73%, in contrast to the nearly total convergence observed with random split or full evidence sharing (Figure 5).

Figure 5: BM25 partitioning sustains more inter-agent diversity after deliberation compared to random allocation and all-public setups.

Implications and Prospects

The principle of designed information asymmetry stands as a necessary and sufficient condition for effective multi-agent deliberation with LLMs. The approach demonstrably converts otherwise redundant multi-round debate into productive collective self-correction—the herding pathologies observed in standard debate settings are structurally mitigated. Moreover, rationale-based communication unlocks information transfer channels absent from numeric-only exchanges, circumventing theoretical limitations imposed by the Data Processing Inequality.

Practically, the findings recommend that any multi-agent LLM framework for reasoning under uncertainty (forecasting, collaborative QA, decision support) must engineer explicit informational diversity across agents. More sophisticated evidence partitioning strategies, adaptive allocation, and dynamically learned aggregation policies offer promising directions.

Risks include potential for adversarial or low-quality private evidence to induce polarization or spurious confidence; thus, future deployments should incorporate source reliability assessments and adversarial robustness measures. PolyGym's design provides a benchmark for ongoing methodological comparison and extension.

Conclusion

Designed information asymmetry is established as a fundamental driver of ensemble LLM forecasting accuracy. The InfoDelphi framework exemplifies the operationalization of this principle, validating it with both theoretical and strong empirical evidence. Broad applicability and open challenges remain in adapting and extending these methodologies for increasingly complex, dynamic multi-agent reasoning systems in diverse and high-stakes domains.