MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMs

Abstract: The widespread adoption of LLMs raises critical concerns about the factual accuracy of their outputs, especially in high-risk domains such as biomedicine, law, and education. Existing evaluation methods for short texts often fail on long-form content due to complex reasoning chains, intertwined perspectives, and cumulative information. To address this, we propose a systematic approach integrating large-scale long-form datasets, multi-agent verification mechanisms, and weighted evaluation metrics. We construct LongHalluQA, a Chinese long-form factuality dataset; and develop MAD-Fact, a debate-based multi-agent verification system. We introduce a fact importance hierarchy to capture the varying significance of claims in long-form texts. Experiments on two benchmarks show that larger LLMs generally maintain higher factual consistency, while domestic models excel on Chinese content. Our work provides a structured framework for evaluating and enhancing factual reliability in long-form LLM outputs, guiding their safe deployment in sensitive domains.

• The paper introduces MAD-Fact, a multi-agent debate framework that decomposes long responses into atomic claims and leverages diverse agent roles for rigorous fact-checking.
• It develops the LongHalluQA benchmark by employing a multi-stage data curation process to ensure high-quality, culturally relevant Chinese long-form factual evaluation.
• Empirical results demonstrate MAD-Fact’s efficacy with an 80% win rate and weighted F1 metrics that closely align with human factuality judgments across various LLMs.

Multi-Agent Debate for Robust Long-Form Factuality Evaluation in LLMs

Motivation and Background

Factuality assessment remains a pivotal challenge for the safe deployment of LLMs, especially in high-stakes domains such as law, biomedicine, and education. Existing factuality benchmarks and evaluation protocols typically emphasize short-form question answering, neglecting the complexities introduced by long-form generation, such as multi-perspective reasoning and the cumulative risk of intertwined factual errors. Standard approaches often rely on single-verifier architectures and treat all atomic claims equally, disregarding claim importance. This paper introduces "MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMs" (2510.22967), proposing an integrated suite of dataset construction, multi-agent verification, and importance-weighted evaluation metrics.

Figure 1: Illustration of core versus auxiliary factual claims; errors on core claims have greater impact on perceived answer quality.

LongHalluQA: Construction of a Chinese Long-Form Factuality Benchmark

Addressing the scarcity of Chinese resources for factuality evaluation, the authors present LongHalluQA, a large-scale, multi-topic benchmark for long-form factual content. The construction follows a rigorous process involving knowledge base curation, multi-round retrieval and verification, semantic clustering, question expansion, and systematic human review to ensure sample consistency and relevance.

Figure 2: The three-stage pipeline for LongHalluQA: knowledge base creation, question generation, and sample quality control.

The dataset targets coverage across domains including Chinese culture, science, and social studies, with an 86% retention rate of high-quality samples after manual vetting.

Figure 3: Visualization of LongHalluQA's topic distribution, highlighting coverage breadth.

LongHalluQA substantially increases answer length and semantic richness compared to its short-form antecedents HalluQA and ChineseSimpleQA.

Figure 4: Comparative examples of responses in HalluQA vs. LongHalluQA, illustrating increased depth and length.

MAD-Fact: Multi-Agent Debate System Architecture

MAD-Fact formalizes long-form factuality evaluation as a three-stage multi-agent interaction: clerk for atomic claim extraction, jury for fact-checking via multi-agent debate, and judge for aggregating verdicts. The clerk decomposes responses into fact-checkable atoms, filtering out non-verifiable or subjective content. The jury, composed of agents with diversified professional roles, debates each claim using direct, retrieval-based, or adaptive strategies. This design supports both autonomous and mandatory evidence retrieval, as well as adaptive continuation or early consensus.

Figure 5: MAD-Fact framework overview, demonstrating the pipeline from atomic claim decomposition through multi-agent debate to final judgment.

Agents interact under three systematically constructed debate protocols (autonomous, mandatory, and dynamic retrieval), balancing retrieval cost, prior knowledge utilization, and reliability.

Figure 6: Illustration of the three debate rule types implemented in MAD-Fact with $N = 3$ agents and two rounds.

Role assignments—including Public, Critic, News Author, Scientist, Psychologist, and Data Analyst—increase epistemic diversity, with incentives provided for external evidence retrieval to counter overconfidence and bias propagation.

Fact Importance Hierarchy and Weighted Evaluation Metrics

The MAD-Fact framework incorporates a hierarchical pyramid-based model for fact importance, mapping atomic claims to frequency-weighted tiers based on their mention across expert reference responses. Claims repeatedly cited receive higher weights, providing non-uniform scoring aligned with human judgment.

Figure 7: Fact importance hierarchy model workflow, aggregating multi-model responses into a weighted pyramid for evaluation.

Weighted precision, recall, and F1 metrics are introduced, accommodating both the claim's factual status and its relative significance, with controlled recall amplification via a tunable $\gamma$ hyperparameter. The metric's correlation with human ratings reaches $r = 0.701, p = 0.036$, validating its reliability.

Figure 8: Pearson correlation between weighted F1 (@$\gamma=0.8$) and human factuality scores.

Empirical Evaluation of MAD-Fact and LLMs

MAD-Fact is systematically evaluated on five fact-checking datasets and two long-form benchmarks (LongFact and LongHalluQA), benchmarking nine LLMs from seven model families.

Figure 9: Factuality evaluation performance of nine selected LLMs on LongFact.

Core findings include:

• Larger, more recent LLMs display superior factual consistency for English long-form generation. GPT-4-Turbo achieves the highest F1 among closed-source models. Domestic models such as Doubao-1.5-Pro demonstrate parity with state-of-the-art international counterparts, diverging from trends observed in short-form QA.
• Chinese-specific LLMs, including QwQ-32B and Doubao-1.5-Pro, outperform international models on the LongHalluQA benchmark, underscoring cultural and linguistic gaps in generic LLM training.

  Figure 10: Factuality evaluation of nine LLMs on LongHalluQA; domestic models lead performance.

• Multi-agent debate with diversified roles and autonomous retrieval reliably surpasses single-verifier baselines (SAFE, FIRE) in overall win rate (MAD-Fact F1 win rate: 80% across label categories and datasets).
• Ablation analyses confirm the necessity of multi-round debate and external search modules; inter-agent misleading increases when models from distinct families are combined, reducing consensus accuracy.

Case Study: Multi-Agent Human-Like Reasoning Behaviors

MAD-Fact agents exhibit nuanced behavior akin to human debate dynamics: adherence to initial beliefs (Figure 11), peer correction (Figure 12), and self-reflection leading to consensus revision (Figure 13).

Figure 11: Case study of agents maintaining individual positions across debate rounds.

Figure 12: Case study where agents collaboratively identify and correct erroneous peer judgments.

Figure 13: Agents engage in recursive self-reflection, shifting consensus after further deliberation.

These behaviors underpin the improved system reliability when compared with rigid single-agent methodologies.

Implications and Future Directions

The MAD-Fact framework robustly addresses the central challenges in long-form LLM factuality assessment: dataset scarcity (especially in Chinese), systematic bias in single-verifier pipelines, and the uniform weighting of atomic claims. The introduction of debate protocols, claim importance weighting, and multi-agent epistemic diversity produces metrics closely aligning with human judgement. This has practical consequences for model selection, deployment, and optimization in sensitive applications, particularly where errors on core claims have outsized impacts.

Further development is warranted for real-world dataset grounding in high-risk domains, optimization of debate efficiency (reducing token/cost overhead), and mitigations against communication hallucinations leading to premature consensus on erroneous claims. Integration of confidence estimations, adversarial training, and adaptive agent weighting in debates could extend robustness.

Conclusion

This work defines a structured framework for long-form factuality evaluation, integrating a new benchmark resource (LongHalluQA), a multi-agent debate system (MAD-Fact), and claim importance-weighted metrics. Empirical analyses reveal robust performance gains, with confirmatory trends in model scaling and cultural specialization. The methodology outlined herein provides a reference foundation for future research and system development targeting reliable long-form LLM assessment, with immediate relevance for high-stakes deployment contexts.