EvidentialRAG: Quantifying and Mitigating Information Conflict in Multi-Source Retrieval-Augmented Generation via Evidential Deep Learning

Abstract: Retrieval-augmented generation grounds LLMs in external evidence, but most pipelines still treat retrieved passages as deterministic and mutually consistent context. In open information environments, retrieved sources may disagree because of temporal drift, source error, ambiguity, or genuine uncertainty. This paper introduces ERAG, an uncertainty-aware RAG framework that converts retrieved chunks into probabilistic evidence before generation. A lightweight evaluator extracts candidate claims and maps chunk-level support to Dirichlet evidence. A conflict-preserving Dempster-Shafer fusion rule then transfers unresolved disagreement into epistemic uncertainty rather than normalizing it away. The generator is routed to direct answering, conflict-aware answering, or abstention according to the fused uncertainty score. Experiments on CRAG, ConflictQA, and MuSiQue show that ERAG remains competitive with the strongest matched baseline on standard question answering while improving behavior under conflict. On the CRAG ambiguous subset, hallucination decreases from 45.3% for Corrective RAG to a human-calibrated estimate of 34.8%, conflict resolution increases from 35.2% to 51.2%, and expected calibration error improves to 0.122. These results suggest that evidential modeling is a practical mechanism for trustworthy information processing in foundation-model-based retrieval systems.

• The paper introduces evidential deep learning to quantify and resolve evidence conflicts in retrieval-augmented generation systems.
• It proposes a fusion operator that explicitly transfers conflict mass to global uncertainty using Dempster-Shafer principles.
• Empirical results demonstrate reduced hallucination and enhanced calibration across benchmarks like CRAG and ConflictQA.

EvidentialRAG: A Conflict-Aware Retrieval-Augmented Generation Paradigm

Motivation and Background

This paper addresses a central issue in retrieval-augmented generation (RAG): the silent assumption of knowledge consistency in retrieved context. LLM-based RAG systems traditionally concatenate retrieved passages and treat them as uniformly reliable, disregarding adversarial evidence quality and inter-source conflict. As RAG is deployed in real-world complex information ecosystems—where data are temporally volatile, error-prone, and often ambiguous—the inability to represent or expose knowledge conflict fundamentally limits reliability and trust calibration.

Recent advances in retrieval architectures and RAG controllers (e.g., late-interaction retrievers, self-reflective pipelines, reranking, and corrective evaluation [Asai2024SelfRAG, Yan2024CRAG, Gao2024RAGSurvey]) have indeed improved the quality of grounding and relevance. Yet, these approaches primarily mitigate retrieval irrelevance, not epistemic uncertainty or conflict propagation within the retrieved evidence set. Empirical findings from benchmarks such as CRAG, ConflictQA, and ConflictBank consistently demonstrate elevated hallucination rates and failure to recognize or communicate unresolved information conflict when standard RAG architectures are applied.

Methodological Contributions

EvidentialRAG introduces a conflict- and uncertainty-aware formulation for RAG, leveraging evidential deep learning (EDL) and Dempster-Shafer theory as a foundation for probabilistic evidence modeling and aggregation.

Evidential Chunk Evaluation

Each retrieved context chunk is semantically processed using a lightweight evidential evaluator, which (1) extracts candidate claim(s), (2) aligns answer candidates across the retrieval set, and (3) generates chunk-level non-negative evidence scores. These vectors parameterize a Dirichlet distribution, with the evidential mass encoding both belief in specific claims and the degree of epistemic uncertainty. This yields a mass-function mapping where singleton masses denote claim-specific support and the frame mass encodes generic ignorance, rectifying the standard misattribution of ignorance as evidence for an artificial unknown singleton.

Candidate Normalization and Alignment

Chunk-extracted claims are normalized and aligned to eliminate surface-form-induced spurious conflict. This includes paraphrase resolution, canonical entity mapping, and task-specific claim disambiguation. The process is conservative and maximizes preservation of true semantic conflicts, ensuring that conflict is not an artifact of superficial variation.

Conflict-Preserving Evidence Fusion

EvidentialRAG applies an iterative conflict-preserving Dempster-Shafer-style fusion operator incorporating a tunable transfer parameter $\lambda$. By eschewing classical Dempster normalization—which overcompresses conflict into overconfident beliefs—the fusion rule explicitly transfers conflict mass into global epistemic uncertainty, making uncertainty quantification structurally robust to irreconcilable evidence. When $\lambda=1$, all conflict is allocated to the frame (Yager’s rule), ensuring the generator does not receive false overconfidence even under strong disagreement. The result is a fused mass function preserving both supporting evidence and unresolved contradictions.

Uncertainty-Guided Routing and Generation

A global uncertainty metric drives a three-action routing policy: direct answer (low uncertainty), conflict-aware answer (moderate conflict), and abstention (high uncertainty). The generator is conditioned on both the fused evidence and the epistemic state, enabling explicit communication of disagreement and principled refusal to answer when warranted.

Empirical Evaluation

Evaluation is conducted across CRAG, ConflictQA, and MuSiQue, capturing open-domain, conflict-rich, and compositional multi-hop settings, respectively. Metrics span exact match, F1, hallucination rate, conflict resolution rate (CRR), calibration error, and abstention behavior.

Key results include:

• On the CRAG ambiguous subset, EvidentialRAG reduces hallucination from 45.3% (Corrective RAG) to 34.8%, raises explicit conflict resolution from 35.2% to 51.2%, and improves expected calibration error to 0.122.
• On ConflictQA’s memory-evidence conflict split, EvidentialRAG achieves a hallucination rate of 38.5% and a refusal F1 of 74.2%—both strongest among the compared systems.
• On MuSiQue, abstention rates are minimized, which demonstrates that conflict-aware abstention does not suppress valid multi-hop synthesis.
• Human audits confirm high-fidelity labeling and robust agreement with LLM-based automated judges, validating the reliability of reported claims.

Analysis and Discussion

EvidentialRAG establishes explicit uncertainty modeling and conflict representation as necessary primitives for trustworthy RAG. The decoupling of retrieval quality from evidential alignment enables a RAG generator to properly distinguish between insufficient, ambiguous, and actively contradictory evidence.

Theoretical implications center on the integration of subjective logic/Evidential Deep Learning with LLM-based generation: the Dirichlet-based evidence representation establishes a rigorous bridge to mass-function aggregation, while Dempster-Shafer fusion with explicit conflict transfer avoids the core pathologies of classical evidence-theoretic combination under high conflict. The routing policy, parametrized by interpretable uncertainty thresholds, formalizes the trade-off between safety (conservatism in the presence of uncertainty) and utility (coverage of confident answers).

Practically, the modularity of the proposed pipeline enables deployment across heterogeneous RAG architectures, with only the evaluator and fusion module requiring adaptation. The evaluator itself can be instantiated with any high-precision LLM or external classifier capable of structured claim extraction, making the architecture flexible and suited for both domain-general and specialized environments. The batched evaluation implementation strategic reduces latency while retaining per-chunk evidence decision granularity.

The sensitivity analyses indicate that the choice of $\lambda$ effectively controls the system’s position on the safety-utility frontier. Full Yager-style transfer is as effective as the validated $(\lambda=0.6)$ default, confirming robustness of the approach to fusion parameterization.

Limitations and Future Prospects

The approach's main dependency is the reliability of the claim extraction evaluator. The present framework uses a modestly-sized instruction-tuned LLM (Llama-3-8B-Instruct), but in high-stakes or adversarial domains, evaluator fine-tuning and additional calibration against local ground-truth may be essential. Benchmark representativeness also remains a challenge: coverage for provenance, source hierarchy, authority attribution, and temporal drift is incomplete relative to many real-world deployments.

Future lines of research may integrate fine-grained source trust, hierarchical evidence fusion, and end-to-end differentiable evidence flow. Extending evidential modeling to include structured knowledge (e.g., KGs), multimodal evidence, or provenance chains would further increase the practical fidelity of the approach. There is also scope for developing more computationally efficient evaluators through model distillation, especially if chunk-level evidence extraction can be approximated with embedding-based contradiction detectors.

Conclusion

EvidentialRAG redefines RAG as an evidential inference process, systematically modeling claim-level support, epistemic uncertainty, and knowledge conflict in the context-prompt-to-generation pipeline. The demonstrated reduction of hallucination, improved explicit conflict exposure, and robust calibration across diverse benchmarks establish the framework as a practical and theoretically sound foundation for next-generation retrieval-based LLMs and information access systems. Explicit uncertainty handling and conflict-aware generation routing are shown to be essential for responsible deployment in open and imperfect information settings.

Reference:

"EvidentialRAG: Quantifying and Mitigating Information Conflict in Multi-Source Retrieval-Augmented Generation via Evidential Deep Learning" (2607.10491)