CiteEval & CiteBench: Citation Evaluation Suite

• CiteEval and CiteBench are evaluation frameworks that rigorously assess citation quality using context-aware attributions and comprehensive retrieval comparisons.
• They employ fine-grained 1–5 Likert scale ratings and targeted edit actions to measure completeness, credibility, and redundancy in citation outputs.
• The suite supports applications across scientific text generation, citation recommendation, and long-context evaluation, driving improvements in retrieval-augmented systems.

CiteEval and CiteBench refer to a family of benchmarks and evaluation frameworks designed to rigorously quantify and analyze the quality of citations in information-seeking systems. They address the need for principled, context-aware, and fine-grained assessment of source attribution—critical for the trustworthiness of Retrieval-Augmented Generation (RAG), scientific citation text generation, and citation recommendation systems. This article synthesizes the design motivations, methodologies, datasets, metrics, key results, and comparative significance of CiteEval and various incarnations of CiteBench, covering their distinct purposes and influence across the citation evaluation landscape.

1. Rationale and Evaluation Paradigms for Citation Quality

The proliferation of LLM-based RAG systems and citation-aware text generation has intensified demands for reliable citation evaluation. Conventional metrics, primarily rooted in Natural Language Inference (NLI), reduce citation assessment to binary or ternary supportiveness judgments between a statement and its cited passages:

$$r_i = \text{NLI}_\phi\bigl(\text{concat}(\mathcal C_i),\,R_i\bigr)$$

These approaches ignore the broader retrieval set $\mathcal S$, overlook the possibility of higher-quality uncited sources, and treat supportiveness as a coarse attribute, failing to capture credibility, completeness, or redundancy. Citation errors—unsupported, incomplete, or misleading attributions—significantly erode user trust and impede information verification (Xu et al., 2 Jun 2025).

To address these deficiencies, CiteEval introduces a principle-driven framework encompassing:

• Full retrieval context comparison: Evaluation is not limited to cited passages but considers all retrieved candidates, penalizing missed higher-authority or more relevant passages.
• Context-aware attributions: Only statements genuinely attributable to retrieval are eligible for citation evaluation, avoiding penalization for model-internal knowledge, query paraphrases, or chain-of-thought inferences.
• Fine-grained multidimensional rating: Citations are scored on a 1–5 Likert scale reflecting completeness, informativeness, mis-attribution, and redundancy, rather than coarse entailment labels.

This principled structure allows for a multidimensional and contextually grounded appraisal of citation behavior in generated outputs (Xu et al., 2 Jun 2025).

2. The CiteEval Framework: Core Principles and Methodology

CiteEval models citation assessment as learning a function:

$$r_i = f_\theta\bigl(\mathcal C_i;\;\mathcal S,\;R,\;Q\bigr)$$

where $R_i$ is a statement, $\mathcal C_i$ the set of cited passages, $\mathcal S$ the entire retrieval set for query $Q$, and $R$ the full generated response (Xu et al., 2 Jun 2025).

Three foundational principles anchor the framework:

1. Comprehensive retrieval comparison: $f_\theta$ explicitly compares $\mathcal C_i$ to $\mathcal S \setminus \mathcal C_i$; citations are penalized if more complete or credible evidence exists in the uncited retrieval set.
2. Explicit context attribution: Each statement is classified into "Query," "Retrieval," "Response," or "Parametric" context. Only "Retrieval" statements are subject to citation rating, ensuring evaluation targets only justified attribution behaviors.
3. Fine-grained rating and edit actions: Annotators use a 1–5 scale with categorical guidelines and perform “critical edits” to citations, selecting from atomic actions (delete-misleading, delete-substandard, delete-redundant, add-evidence, add-refinement, add-credibility). This operationalizes nuanced feedback and supports both exhaustive ("Full") or citation-restricted ("Cited") evaluation scenarios.

CiteEval’s annotation reliability is high (Krippendorff’s α: 0.980 for context attribution, 0.774 for citation rating). Approximately 87% of sentences in its benchmark are annotated as "Retrieval" (Applicable), with a substantial fraction requiring critical edits, revealing the nontriviality of citation correctness in practical LLM outputs (Xu et al., 2 Jun 2025).

3. CiteBench Benchmarks: Design, Data, and Diagnostic Axes

CiteBench is an umbrella term for multiple benchmarks, each addressing different facets of citation modeling and evaluation. The key variants are:

Benchmark	Primary Purpose	Domain/Task Coverage	Key Evaluation Dimensions
CiteBench (2022)	Textual citation generation	Scientific writing	ROUGE, BERTScore, citation intent
CiteBench (2024)	Citation recommendation performance	Local recommendation	Recall/MRR@10, eight diagnostics
CiteBench (2025)	Principle-driven citation quality	RAG/long-form QA	Human 1–5 rating, critical edits

3.1 CiteBench for Citation Text Generation (Funkquist et al., 2022)

This incarnation brings together datasets (ABURAED, XING, LU, CHEN Delve/S2ORC) spanning computational linguistics and multiple sciences, unifying citation text generation as: generate a citation text $T'$ conditioned on cited document set and citing context, evaluated via ROUGE, BERTScore, and citation-intent/CORWA tagging.

Models include extractive (LEAD, TextRank, LexRank) and abstractive (Longformer Encoder-Decoder, LED) baselines. Domain-specific fine-tuning yields best scores, and transfer learning works best between input/output format-matched splits. Human evaluation reveals moderate readability and limited factual consistency (no model exceeds 3.0 on a 5-point consistency scale), underscoring a gap in accurate synthesis from sources.

3.2 CiteBench for Citation Recommendation (Maharjan, 2024)

This variant is tailored for context-aware citation recommendation. Built from S2ORC and S2AG, it provides millions of local context-citation pairs across 19 fields with detailed diagnostic slices: field of study, year, cited-paper popularity, context length, citation location, intent, part-of-speech, and low-resource domains. Evaluation requires models to rank the correct cited paper from the entire candidate pool, quantified via Recall@K, MRR, MAP, and nDCG@K.

Most learned models (NCN, LCR, Galactica) are currently outperformed by the BM25 baseline, but specific models show slice-specific strengths. The benchmark enables reproducible, granular understanding of strengths and weaknesses across axes highly relevant for citation recommendation system design and optimization.

3.3 CiteBench for Principle-Driven Citation Evaluation (Xu et al., 2 Jun 2025)

This benchmark is tightly aligned with CiteEval’s framework, featuring multi-domain, statement-level, triple-blind annotated datasets. Queries are drawn from ASQA, ELI5, MS MARCO, and LFRQA, with responses generated by diverse LLMs and annotated with fine-grained ratings and critical edits. Splits are provided for both metric development and held-out evaluation. This configuration enables both tuning and robust meta-evaluation of automatic citation evaluation models.

4. Automated Metrics and Experimental Results

To scale evaluation, CiteEval introduces CiteEval-Auto, leveraging LLMs and regression-based scoring:

• Context attribution step: GPT-4o assigns context classes to statements ($F_1=0.957$ distinguishing Applicability).
• Citation scoring: Two primary approaches, Iterative Chain-of-Edits (IterCoE) and Edit Distance (EditDist), are used. IterCoE elicits LLM rationales for editing citation sets and assigns normalized scores; EditDist weights edit actions by regression distances fitted to human scores.
• Aggregation and masking: Only "Applicable" (Retrieval) statements contribute to pooled citation quality metrics.

CiteEval-Auto achieves strong correlation with human scores (e.g., Pearson = 0.731, Spearman = 0.559 at statement level), outperforming AutoAIS, AttriScore, and LQAC across both statement and response levels. Ablation experiments confirm the necessity of each principle: omitting context attribution or edit-based reasoning leads to significant drops in correlation (Xu et al., 2 Jun 2025).

Key findings:

• CiteEval-Auto delivers highest consistency with human judgments across varied RAG and LLM systems.
• In the Cited scenario, GPT-4o attains the best performance; in the Full scenario (with missing citation penalization), Llama-3-70b slightly leads.
• Improved retrieval recall correlates with better citation quality; aggressively filtering for precision produces variable results.
• Iterative application of edit-based feedback enables small models to improve citation quality, reducing the gap with larger models.
• Longer responses are associated with higher missing-citation ratios and, consequently, lower citation quality.

5. Comparative View: L-CiteEval and Other Citation Benchmarks

L-CiteEval (Tang et al., 2024) expands the citation evaluation landscape to long-document contexts (up to 48K tokens) and encompasses eleven tasks (QA, summarization, dialogue, synthetic reasoning) spanning diverse domains. Unlike CiteBench (short-passage, scientific focus) and faithfulness schemes that rely on external LLM evaluators, L-CiteEval utilizes automated, NLI-based citation verification, with fully scripted context extension, chunking, and evaluation processes.

Notably, closed-source models (GPT-4o, Claude-3.5-Sonnet) achieve higher citation recall (CR), precision (CP), and F1, especially on multi-hop or reasoning-heavy tasks, than open-source counterparts. RAG pipelines dramatically improve open-source LCM faithfulness at a modest cost to output fluency. A strong alignment is observed between LCM attention mechanisms and cited text spans, validating citation presence as a plausible diagnostic for context adherence.

CiteBench (in all incarnations) fills crucial gaps:

• Text generation: Reference-based overlap and discourse metrics to evaluate citation synthesis (Funkquist et al., 2022).
• Citation recommendation: Global comparison with feature-based diagnostic clarity (Maharjan, 2024).
• Principle-driven RAG citation quality: Fine-grained, human-validated scoring and LLM-driven meta-evaluation (Xu et al., 2 Jun 2025).
• Long-context citation faithfulness: Multi-domain, fully automated, large-context benchmarks (Tang et al., 2024).

6. Future Directions and Limitations

Identified limitations and future directions include:

• Extension of context attribution taxonomies, e.g., personalization contexts,
• Distillation of evaluation models for low-compute applications,
• Integration with user studies to assess metrics’ impact on trust and verification,
• Expansion towards chunk-level, joint retrieval-citation evaluation,
• Enhanced support for global recommendation tasks and multi-lingual, multi-modal scenarios,
• Development of reference-grounded factuality metrics beyond overlap-based and NLI proxies.

A plausible implication is that, as systems increasingly automate source attribution in high-stakes environments, robust, multi-principle evaluation frameworks like CiteEval—grounded in diagnostic benchmarking (CiteBench)—will become essential for both research progress and safe real-world deployment.

7. Comparative Table: CiteBench and CiteEval Contexts

Framework/Benchmark	Primary Task	Evaluation Signature	Distinctive Features
CiteEval + CiteBench (2025)	RAG citation attribution	Human 1–5 Likert, LLM auto-metrics	Full retrieval, context attribution
CiteBench (2022)	Citation text generation	ROUGE, BERTScore, intent/CORWA	Unified scientific datasets, qualitative diagnostics
CiteBench (2024)	Local citation recommendation	Recall@10, MRR, MAP, nDCG@K	Diagnostic slices, large candidate pool
L-CiteEval (2024)	LCM-long-context citation faithfulness	Citation P/R/F1, Rouge, attention trace	8-48K token context, concrete automated metrics

Overall, CiteEval and CiteBench together define a comprehensive, principle-driven, and diagnostic suite for evaluating and improving the faithfulness, accuracy, and interpretability of citation behavior in both generative models and recommendation systems. Their design and empirical results substantiate a new standard for citation evaluation, steering both research and practice toward higher verifiability and trust in information-seeking systems (Xu et al., 2 Jun 2025, Funkquist et al., 2022, Maharjan, 2024, Tang et al., 2024).