ChatEval: Multi-Agent NLG Evaluation

• ChatEval is a multi-agent framework that employs diverse LLMs to debate and score NLG responses, simulating a human panel evaluation process.
• It uses distinct communication strategies and role-specific prompts to generate comparative judgments and scalar scores, enhancing evaluation precision.
• Empirical results show that ChatEval improves alignment with human preferences through increased accuracy and inter-annotator agreement across benchmarks.

ChatEval is a multi-agent debate framework for automating the evaluation of natural language generation (NLG) systems using LLMs as evaluators. It replaces conventional single-agent LLM-based judgment with teams of simulated “referees” that discuss, critique, and ultimately score or rank candidate responses. This design draws from best practices in human assessment—where diverse annotator teams deliberate—thereby increasing alignment with human preferences, mitigating groupthink, and modeling real-world evaluation complexity (Chan et al., 2023).

1. System Architecture and Debate Protocols

ChatEval orchestrates N LLM-based agents (e.g., GPT-4 or GPT-3.5-turbo; N=2–4 by default), each initialized with a distinct persona (“role prompt”) such as Critic, News Author, Psychologist, Scientist, or General Public. This diversity compels agents to approach evaluations from varied analytical standpoints.

Agents engage in a multi-round chat (typically T=2 rounds) according to one of three communication strategies:

• One-by-One: Agents respond in strict sequence; each new utterance is appended to the subsequent agents’ local chat history.
• Simultaneous-Talk: All agents generate responses in parallel for each round; all replies are broadcast to all agents.
• Simultaneous-Talk-with-Summarizer: Simultaneous generation as above, but each round’s responses are summarized by a separate LLM and the summary is inserted as shared history.

The interaction is formalized as follows. Let $H_n$ be the chat history for agent $n$; $D_n(\cdot)$ denotes the LLM instance with the respective role prompt:

• One-by-One:

$$h^{(t)}_n \leftarrow D_n(H_n);\quad \text{for all}\ m>n: H_m \gets H_m \,\|\, h^{(t)}_n$$

• Simultaneous Talk:

All agents produce $h^{(t)}_{1..N}$ in parallel, buffer $B=\{h^{(t)}_n\}$, then each $H_n \gets H_n \,\|\, B$.

• Summarizer Variant:

$H_n \gets H_n \,\|\, \mathrm{SUM}(B)$ for summarizer LLM $\mathrm{SUM}$.

After T rounds, agents independently issue final scores (scalar or comparative). There is no explicit consensus step.

2. Mathematical Decision Rules and Output Formats

ChatEval supports two primary evaluation modes:

• Comparative Judgments (“Which assistant’s answer is better?”): Each agent picks a candidate; majority vote decides the final winner:

$$j^* = \arg\max_{j\in\{1,2\}} \sum_{n=1}^N \mathbb{1}[\text{agent}_n^\text{choice} = j]$$

• Scalar Scoring (range 1–10): Each agent issues a numeric score $s_n$; the system reports the mean:

$S = \frac{1}{N} \sum_{n=1}^N s_n$

Correlation with human preferences is quantified using task-appropriate metrics: accuracy and Cohen’s κ for open-ended QA; Spearman’s ρ and Kendall’s τ for dialogue response rankings. No statistical significance testing (e.g., $p$-values) is performed; gains are supported by consistency across tasks and metrics (Chan et al., 2023).

3. Experimental Methodology and Baselines

Prompt templates give each agent access to the context (source_text), candidate responses, its role description, and an accumulating chat_history slot. Agents are “primed” to make their utterances brief and focused.

Initialization sets agent histories with the system message, role, and all inputs. The debate proceeds for a fixed T (typically T=2). Conflict resolution relies on unweighted majorities or mean scores; ties in comparisons default to “equal quality.”

Experiments use:

• Datasets:
  • FairEval (open-ended QA, 80 questions, human-labeled majority votes)
  • Topical-Chat (dialogue, 60 contexts, human judgments on four aspects)
• Models and Baselines:
  • ChatEval MA (GPT-4 or GPT-3.5-turbo, temperature=0)
  • Single-agent LLM (identically prompted, but without roles/debate)
  • FairEval (MEC+BPC ensembles)
  • G-EVAL (CoT+probability-weighted sum)
  • Traditional automatic metrics (BLEU-4, ROUGE-L, BERTScore)

Performance is assessed by comparing model output to human reference rankings or scores.

4. Empirical Results and Ablation Studies

Quantitative Outcomes

• Open-ended QA (FairEval):
  • Human: Acc = 71.7% (mean), κ ≈ 0.54
  • ChatEval MA (GPT-4): Acc = 63.8%, κ = 0.40
  • Single-agent GPT-4: Acc = 61.3%, κ = 0.36
  • ChatEval (GPT-3.5-turbo): Acc = 60.0%, κ = 0.33; Single-agent: 53.8%, κ = 0.27
  • Comparative baselines (FairEval MEC+BPC) outperform single-agent LLMs but are matched or exceeded by ChatEval (Chan et al., 2023).
• Dialogue (Topical-Chat):
  • Traditional metrics (BLEU, ROUGE, BERTScore): all ρ < 0.35
  • G-EVAL-4: ρ = 0.588, τ = 0.575
  • Single-agent GPT-4: ρ = 0.658, τ = 0.611
  • ChatEval GPT-4 MA: ρ = 0.684, τ = 0.632

Ablation Insights

• Diverse Role Prompts are essential: Multi-agent with role diversity (ChatGPT: 60.0%) far outperforms agents with identical generic roles (53.8%).
• Communication Strategy: Strict turn order (one-by-one) gives the best results; simultaneous talk and summarizer protocols yield lower gains.
• Number of Agents: Accuracy climbs from N=2 → N=4 and then declines at N=5; inter-annotator agreement κ increases with N, up to N=4.
• Debate Rounds: Beyond T=2, additional context dilutes focus and does not improve reliability.

Qualitative Observations

Agents demonstrate human-like debate phenomena: opening statements, stance challenges, alternative proposals, and eventual convergence on a verdict—often matching human consensus on ambiguous cases (Chan et al., 2023).

5. Authority Dynamics and Social Power Effects

Subsequent analysis reveals that explicit assignment of “authority” roles within ChatEval debates—classified per French & Raven’s power theory (Legitimate, Expert, Referent)—induces significant alignment shifts among “general” agent counterparts (Choi et al., 8 Jan 2026). Specifically:

• Expert and Referent power roles exert stronger pull (ΔA ≈ 7.2–7.5%) over General Public agents than do Legitimate power roles (ΔA ≈ 2.8%).
• Mechanism: The authority agent maintains consistent positions after declaring its preference; the general agent is comparatively flexible, altering decisions toward the authority’s stance—even in content-controlled, “role-label only” settings.
• Bias Dynamics: Authority-induced agreement emerges rapidly (by turn 4 of 12) and then stabilizes.
• Mitigation: Rotating or randomizing role labels, explicit challenge mechanisms, and balancing authority types are recommended for fair multi-agent debates (Choi et al., 8 Jan 2026).

6. Comparative Approaches: LLM-Based Turn-Level Evaluators

Alternative single-agent LLM evaluation paradigms, as employed in “Three Ways of Using LLMs to Evaluate Chat,” illustrate the limitations of unmoderated prompting for granular, turn-level judgment (Plátek et al., 2023):

• Simple Open LLM Prompting: Extremely fragile to response format, producing invalid outputs in ∼50% of cases; negligible test-retrieval correlation (ρ = 0.0807).
• Feed-Forward Regressor on LLM Embeddings: Slight improvement (ρ = 0.1742), but normalization across datasets degrades reliability.
• Dynamic Few-Shot Retrieval with ChatGPT: Prompting with two nearest dev examples (per MPNet/FAISS similarity search) substantially improves performance (ρ = 0.4190; best component: appropriateness ρ = 0.488), though at high computational cost.

Ablations indicate that ChatGPT benefits disproportionately from few-shot prompting, while open-source models such as Llama 2 do not; prompt engineering and sample normalization are highly consequential (Plátek et al., 2023).

7. Applications, Limitations, and Future Directions

ChatEval is openly available with architecture, prompt templates, and variant debate protocols for reproducibility and further research (Chan et al., 2023).

Applications

• Human-aligned evaluation of generative AI models for open-ended QA and multi-dimensional dialogue assessment.
• Studying group decision processes and social dynamics within LLM-based multi-agent systems.
• Exploring authority, consensus, and fairness phenomena in automated evaluation settings (Choi et al., 8 Jan 2026).

Limitations

• Homogeneous agent populations; influence of mixing LLM types remains untested.
• Cost and latency scale linearly with the product $N \times T$.
• No formal statistical testing of performance improvements is reported.

Future Enhancements

• Heterogeneous agent teams with LLMs of varied strength/family.
• Adaptive debate termination based on agent convergence.
• Learned agent weighting schemes, integrating past reliability.
• Integration of human-in-the-loop for adjudication or training.
• More advanced conflict-resolution protocols (e.g., tournament voting, rebuttal rounds).

A plausible implication is that as LLM-based evaluation matures, frameworks like ChatEval will provide both research infrastructure for systematic protocol development and testbeds for understanding the social dynamics emergent in artificial judge teams, thus informing both NLG evaluation and broader multi-agent AI system design (Chan et al., 2023, Choi et al., 8 Jan 2026).