Paragraph-Level Relevance Annotation

• Paragraph-level relevance annotation is a technique that labels paragraphs with relevance scores or binary masks to extract focused evidence from multi-paragraph documents.
• It employs similarity-based approaches, neural QA models with attention, and reinforcement learning to optimize sparsity, continuity, and calibration in annotations.
• Empirical results show marked improvements in retrieval and explanation fidelity, with significant F1 score gains in multi-paragraph QA and legal reasoning tasks.

Paragraph-level relevance annotation denotes the task of identifying, extracting, or assigning relevance scores or binary labels to individual paragraphs within multi-paragraph documents, with the aim of supporting complex information retrieval, question answering, legal reasoning, explainable decision-making, or multimodal analysis. Compared to word- or sentence-level annotation, paragraph-level annotation operates at a coarser but semantically coherent granularity, facilitating both interpretability and effective content selection across domains such as legal case law, deep learning-based reading comprehension, multimodal document analysis, and legal information retrieval.

1. Formalization and Annotation Protocols

Paragraph-level relevance is typically cast as a task in which a document $D = [P_1, ..., P_N]$ is partitioned into $N$ paragraphs. Each paragraph $P_i$ may be assigned a binary relevance label $z_i \in \{0,1\}$ (e.g., rationale mask), a confidence score (e.g., logit scores), or included in a ranked or clustered output, depending on application (Chalkidis et al., 2021). In multi-label settings, as in legal judgment prediction, the output comprises both a vector of target labels $Y \in \{0,1\}^{|A|}$ and paragraph-wise binary mask $Z=[z_1,...,z_N]$ denoting the rationale sufficient to recover $Y$.

Annotation strategies vary. Manual expert annotation is costly and rarely scalable; alternatively, "silver" rationales are derived from court citations or LLM outputs, with groupings or sparse selections serving as proxies for human-labeled evidence (Chalkidis et al., 2021, Nguyen et al., 30 Sep 2025). In deepfake detection, large-scale paragraph-level rationale annotation is achieved via an LLM pipeline that clusters predefined forensic features into paragraphs and aligns explanations with global classification (Nguyen et al., 30 Sep 2025). In retrieval contexts, explicit annotation may be bypassed entirely; instead, external structures (e.g., citation graphs) act as weak ground-truth for relevance (Sisodiya et al., 2023).

2. Methodologies for Paragraph-Level Relevance Scoring

Several algorithmic paradigms have been developed:

• Similarity-Driven Approaches: Paragraphs are embedded via TF-IDF, bag-of-words, or word-level embeddings (e.g., Word2Vec). Inter-document relevance relies on maximum or average paragraph-pair similarity, aggregated via "Mean-Paragraph" or "Fixed-Paragraph" methods (Sisodiya et al., 2023).
• Neural QA and Rationale Extraction: For question answering and interpretability, paragraphs are passed through shared neural encoders (e.g., BERT), contextualizers, and rationale scorers. Hard or soft attention masks select relevant paragraphs according to sparsity and comprehensiveness constraints (Chalkidis et al., 2021).
• Policy Optimization at the Paragraph Level: In vision-language tasks, explanation generation is modeled as a sequence of paragraph-level actions, with reinforcement learning (e.g., PRPO) deployed to optimize output quality and grounding (Nguyen et al., 30 Sep 2025).
• Shared-Normalization Training: In deep document QA, calibration across multiple paragraphs is enforced by a shared softmax denominator—forcing the model to produce globally comparable confidence scores and discouraging overconfident assignment to a single paragraph (Clark et al., 2017).

3. Constraints, Regularization, and Calibration

Paragraph-level annotation models are enhanced by domain- and task-specific constraints:

• Sparsity ($L_s$): Encourages selectivity, limiting the proportion of paragraphs labeled as relevant, matching the empirical sparsity from ground-truth rationales (Chalkidis et al., 2021).
• Continuity ($L_c$): Penalizes non-contiguous selection; however, this may harm performance at paragraph granularity and is often disabled in legal settings (Chalkidis et al., 2021).
• Comprehensiveness ($L_g$): Enforces that information omitted by $Z$ (the complement mask $Z^c$) is uninformative for the classification task, with representation-, probability-, or loss-based measures (Chalkidis et al., 2021).
• Singularity ($L_r$): Ensures that $Z$ not only outperforms its complement but also random masks of equivalent sparsity, improving rationale precision (Chalkidis et al., 2021).
• Calibration via Shared-Norm or No-Answer Objective: Shared normalization across paragraphs yields logits that reflect global evidence, essential for robust inference under multi-paragraph inputs (Clark et al., 2017). A per-paragraph "no-answer" head can abstain when confidence is insufficient.

Reward designs in paragraph-level RL, such as PRPO, combine visual consistency (keyword-image similarity) and classification consistency (agreement with preceding paragraph labels) for dense credit assignment at the paragraph level (Nguyen et al., 30 Sep 2025).

4. Practical Pipelines and Integration into Downstream Tasks

The operationalization of paragraph-level annotation varies by domain:

Domain	Input/Output	Paragraph Relevance Mechanism
Legal Precedence Retrieval	(Judgment, Paragraphs)	Cosine similarity aggregation (PL-M, PL-F), citation graph as implicit supervision (Sisodiya et al., 2023)
Question Answering	(Question, Doc Paragraphs)	Paragraph sampling via TF-IDF, neural reader (BiDAF + self-attention), shared-normalization (Clark et al., 2017)
Legal Rationale Extraction	(Case Facts, Labels)	Hierarchical BERT encoder, rationale extractor, regularized hard-masking (Chalkidis et al., 2021)
Multimodal Deepfake Detection	(Image, Reasoning Paragraphs)	LLM feature discovery, paragraph clustering, RL-based PRPO for explanation optimization (Nguyen et al., 30 Sep 2025)

These pipelines share core steps: document segmentation, paragraph-level feature extraction/ranking, candidate paragraph selection, model inference with cross-paragraph calibration, and, when possible, end-to-end rationale extraction via regularized learning or RL.

5. Evaluation Frameworks and Metrics

Evaluation is multi-faceted, depending on whether the task is retrieval, classification, explanation alignment, or sufficiency:

• Retrieval Metrics: Precision@K, Recall@K, Mean Reciprocal Rank (MRR), Mean Average Precision (MAP), BPREF, all adapted to paragraph-level granularity (Sisodiya et al., 2023).
• Classification and Faithfulness: Multi-label micro-F1, sufficiency (drop in class probability when restricted to selected paragraphs), comprehensiveness (drop when only the complement is available) (Chalkidis et al., 2021).
• Rationale Quality: Precision, recall, F1-score of selected versus gold-standard paragraphs; mean R-Precision (mRP) computed for top-k confident paragraphs (Chalkidis et al., 2021).
• Explanation Quality in Multimodal Contexts: Human or LLM-based rubric scores on evidence grounding, alignment, clarity, and reasoning; composite “reasoning score” aggregates these dimensions (Nguyen et al., 30 Sep 2025).
• Calibration and Overconfidence: Empirical results indicate that without proper cross-paragraph normalization, models degrade in accuracy as the number of distractor paragraphs increases (e.g., drop in F1 beyond $K\approx4$ for document QA without shared-norm) (Clark et al., 2017).

6. Empirical Findings and Pitfalls

Paragraph-level annotation offers stronger discriminative power and interpretability than document-level methods. For legal retrieval, mean and top-k paragraph similarities (PL-M, PL-F) closely follow citation ground truth and outperform document-level baselines (e.g., MAP improvements up to +114%) (Sisodiya et al., 2023). In multi-paragraph QA, shared-normalization and paragraph sampling yield an F1 improvement of +15 points (56.7→71.3) on TriviaQA web compared to the prior best (Clark et al., 2017).

However, error analysis reveals several challenges:

• Continuity constraints may be detrimental at paragraph level, unlike word-level rationale extraction (Chalkidis et al., 2021).
• Silver and LLM-derived rationales may omit critical information or misclassify support, illustrating the limitations of automatic annotation (Chalkidis et al., 2021, Nguyen et al., 30 Sep 2025).
• In open QA and law, calibration is critical; overconfident scoring in irrelevant paragraphs degrades performance (Clark et al., 2017).
• Satisfying multi-label comprehensiveness is more complex than for binary tasks, necessitating new regularization techniques (Chalkidis et al., 2021).

7. Open Challenges and Research Directions

Persistent issues in paragraph-level annotation include the scarcity and noise of annotated rationales (especially gold/human labels), the need for improved regularization tailored to paragraph granularity and multi-label logic, and the development of annotation and credit assignment protocols that generalize across domains. Future research directions highlighted in the literature include:

• Design of new constraints and regularizers specific to paragraphs (addressing multi-label logic, sparsity, modular evidence) (Chalkidis et al., 2021).
• Leveraging silver rationales in semi-supervised, ranking, or curriculum learning settings, and the development of de-biasing/counterfactual methods to reduce spurious patterns (Chalkidis et al., 2021).
• Extending test-time paragraph-level RL (as in PRPO) to other structured tasks in multimodal learning, such as VQA or medical image analysis, where label-free rewards and modular design are critical (Nguyen et al., 30 Sep 2025).
• Incorporation of contextualized embeddings (e.g., Legal-BERT) and supervised neural ranking architectures for improved retrieval and explanation (Sisodiya et al., 2023).
• Systematic collection and validation of human-annotated paragraph-level ground truth to calibrate and benchmark automated systems, including the measurement of inter-annotator agreement.

Empirical evidence suggests that paragraph-level annotation—supported by appropriate modeling, constraint design, and evaluation—yields consistent improvements in discriminative power, explanation faithfulness, and overall task performance across diverse information retrieval, reading comprehension, and reasoning domains (Clark et al., 2017, Chalkidis et al., 2021, Sisodiya et al., 2023, Nguyen et al., 30 Sep 2025).