RebuttalBench: Benchmark for Evidence-Aligned Rebuttals

• RebuttalBench is a comprehensive benchmark suite designed to evaluate evidence-aligned rebuttal generation by decomposing reviewer feedback into atomic concerns.
• It combines a large-scale corpus of review-response pairs with multi-agent evaluation protocols to enhance transparency, plan validation, and evidence anchoring.
• Empirical findings demonstrate that structured hybrid contexts and external retrieval significantly improve coverage, specificity, and overall rebuttal quality.

RebuttalBench is a comprehensive benchmark suite, corpus, and evaluation protocol for the transparent and evidence-aligned generation of peer review rebuttals using multi-agent and evidence-centric natural language processing systems. Designed primarily for the assessment and development of RebuttalAgent architectures, RebuttalBench establishes high-fidelity standards for faithfulness, coverage, and argumentative quality in the context of author responses to scientific peer review. The benchmark reflects the increasing complexity and high-impact requirements of the rebuttal-writing process in venues such as ICLR and NeurIPS, where the quality of rebuttals can critically influence the review process and acceptance outcomes (Ma et al., 20 Jan 2026).

1. Purpose and Motivation

RebuttalBench addresses the shortcomings of direct-to-text generation approaches that struggle with hallucination, incomplete coverage, poor grounding in manuscript evidence, and inadequate treatment of reviewer intent. The benchmark reframes rebuttal generation as a task requiring (i) systematic decomposition of reviewer feedback into atomic concerns, (ii) strategic planning with explicit evidence anchoring, (iii) transparent draft composition, and (iv) rigorous, multi-dimensional evaluation. This approach was necessitated by evidence that LLM-based responses often fail in coverage, faithfulness, or specificity, with direct implications for academic decision-making (Ma et al., 20 Jan 2026).

2. Dataset Construction and Structure

RebuttalBench comprises two main components:

• RebuttalBench-Corpus: A large-scale repository of 9,300+ author response–review pairs mined from ICLR OpenReview, indexed at the level of review segments, author rebuttals, and paper metadata. Each pair is linked at paragraph granularity to enable targeted evaluation.
• RebuttalBench-Challenge: A high-difficulty subset featuring 20 papers, each paired with at least 100 follow-up signals and balanced distributions of positive and negative outcomes. This set targets complex or contentious reviews and further includes multi-turn rebuttal scenarios.

Each sample in RebuttalBench is annotated along distinct axes, enabling both automatic and human-centered evaluation. The benchmark includes not only the ground-truth responses but also a rich set of reviewer comments and paper contexts, supporting evidence-based planning within next-generation RebuttalAgent pipelines.

3. Methodological Principles

RebuttalBench enforces a set of methodological principles for both system development and evaluation:

1. Decomposition into Atomic Concerns: Reviewer feedback is systematically parsed into minimal actionable units, using coverage-maximizing clustering and merging/splitting heuristics. Let $C$ denote raw review segments, and $Q = \{q_i\}$ the atomic concerns. Decomposition maximizes coverage while minimizing redundancy, governed by an objective function

$$\max_{Q}\;\sum_{q\in Q}\sum_{r\,\in\,C}\mathbf{1}\bigl[\mathrm{sim}(q,r)\ge\tau\bigr]\;-\;\lambda\sum_{q\neq q'}\mathrm{sim}(q,q')\,.$$

All operations are validated by a coverage checker (Ma et al., 20 Jan 2026).

2. Hybrid Context Construction: To balance compressibility and high-fidelity evidence grounding, responses are composed from hybrid contexts $H_i$ consisting of compressed summaries for context and selectively restored high-resolution raw text for the most relevant segments; formally,

$$H_i = \bigl(S\setminus\{s_{j_1},\dots,s_{j_k}\}\bigr)\;\cup\;\{r_{j_1},\dots,r_{j_k}\}\,,$$

with $S$ the set of summaries and $\{r_j\}$ the corresponding retrieved paragraphs.

1. Evidence-Centric Response Planning: Before drafting, a RebuttalAgent generates a verifiable plan $P$ specifying for each concern $q_i$ the intended argumentative move (e.g., “Defend” using internal evidence, or “Act” by recommending further experimentation) and the evidence items (internal or externally retrieved) to be cited. Plan completeness and faithfulness are checked before proceeding to draft (Ma et al., 20 Jan 2026).
2. On-Demand External Retrieval: When internal context is insufficient, the benchmark requires use of external retrieval modules (e.g., ArXiv API, literature search). Integration is scored both in terms of raw retrieval correctness and the ability to generate citation-ready evidence summaries (Ma et al., 20 Jan 2026).

4. Evaluation Protocol and Metrics

RebuttalBench prescribes automatic and LLM-as-judge evaluation on several axes:

Metric	Definition	Scoring Range
Coverage	Proportion of reviewer concerns responded to	[1, 5]
Sem-Align	Semantic alignment between response and reviewer concern	[1, 5]
Specificity	Degree to which response is tailored to concern (avoids generic/boilerplate replies)	[1, 5]
LogicConsist	Logical consistency and argumentative soundness	[1, 5]
EvidSupport	Faithful citation or quotation of manuscript or external evidence	[1, 5]
RespEngagement	Direct engagement with reviewer’s question or concern	[1, 5]
ProfTone	Professional tone and manner	[1, 5]
Clarity	Readability and clarity	[1, 5]
Construct	Relative constructiveness of the rebuttal (e.g., proposing concrete revisions)	[1, 5]
Average	Unweighted mean of previous metrics	[1, 5]

Aggregate R-Score (Relevance), A-Score (Argumentation), and C-Score (Communication) are defined as: $$R = \tfrac{R_1+R_2+R_3}{3},\quad A = \tfrac{A_1+A_2+A_3}{3},\quad C = \tfrac{C_1+C_2+C_3}{3},\quad \text{Overall} = \frac{R + A + C}{3}$$

Benchmarks report mean scores per model and per backbone, enabling direct performance comparison across LLMs and pipelines (Ma et al., 20 Jan 2026).

5. Systemic Impact and Empirical Findings

Evaluation across leading LLMs (DeepSeek, Grok, GPT5-mini) reveals that integration with RebuttalAgent—mandating evidence-centric planning, atomic concern decomposition, and checkers for plan completeness—substantially improves all evaluated axes relative to direct prompting. Coverage and specificity see improvements up to +0.78 and +1.33, respectively. The introduction of hybrid context and structured plans provides robust gains in faithfulness (EvidSupport, LogicConsist), with ablation demonstrating that omitting external evidence has the largest negative effect on performance (Ma et al., 20 Jan 2026).

Empirical findings establish that:

• Faithfully cited and plan-checked responses consistently outperform even advanced direct-to-text systems across relevance, logic, and constructiveness.
• Removal of external evidence yields substantial drops in both coverage and constructiveness, affirming that evidence-centric construction is central to rebuttal quality.
• Structured pipelines mitigate hallucination, boilerplate, and omission errors prevalent in earlier systems.

6. Methodological Innovations and Relationship to Prior Work

RebuttalBench operationalizes several innovations:

• Multi-agent, sequential planning model: Distinct "agents" (parser, extractor, strategist, drafter, checker) correspond to modular LLM calls or fused LLM+rule-based logic, supporting compositionality and transparency.
• Hybrid context: Dynamic retrieval of high-resolution passages eliminates fidelity–readability tradeoff.
• Verification-first workflow: Plans are constructed and validated before generation, enabling fine-grained tracing and error localization.
• Interleaved external evidence: Systematic API-driven external lookup resolves concerns unanswerable from the manuscript itself.

RebuttalBench extends task settings from claim detection (Lavee et al., 2019), general-purpose rebuttal argument retrieval (Orbach et al., 2019), and persuasion-centric multi-agent debates (Han et al., 10 Nov 2025) to the peer review context, imposing stricter adversarial and transparency constraints.

7. Limitations and Future Directions

Known limitations include:

• Authoritative performance is presently benchmarked in the context of ICLR/NeurIPS-style reviews; adaptation to other disciplines or less-structured venues requires further data curation.
• Human-in-the-loop components (plan validation, checker design) remain essential steps for production use, given the continuing challenge of hallucination in large generative models.
• While RebuttalBench mandates evidence traceability, the degree to which it mitigates subtle misunderstandings of reviewer intent is a subject of ongoing research.

Priorities for future development include extending the corpus to cover additional venues, refining fine-grained coverage metrics for multi-turn rebuttals, and benchmarking models under mixed-review and cross-paper scenarios (Ma et al., 20 Jan 2026).

---

RebuttalBench has become the principal academic resource for the systematic evaluation of evidence-aligned, transparent, and faithful author response systems in scientific peer review, providing both the dataset and the rigorous multi-dimensional metrics necessary for the assessment of increasingly complex RebuttalAgents (Ma et al., 20 Jan 2026).