Disagreement-Aware NLP Methods

• Disagreement-aware NLP methods are techniques that leverage full annotator label distributions to quantify subjectivity and ambiguity in language tasks.
• They integrate annotator metadata and soft-label training, enhancing model calibration and subgroup fairness across varied human perspectives.
• These methods employ multi-task, embedding, and mixture-of-experts architectures with distributional metrics to robustly capture inherent annotation disagreement.

Disagreement-aware NLP methods constitute a set of modeling, annotation, and evaluation paradigms designed to explicitly represent, learn from, and quantify inter-annotator variation on subjective language tasks. These methods reject the majority-vote approach that collapses annotation distributions into single "ground truth" labels and instead treat disagreement as a primary signal reflecting diverse human perspectives, signal ambiguity, and context-dependent interpretation. By leveraging empirical label distributions, annotator profiles, demographic metadata, and structured pooling architectures, disagreement-aware models aim both to faithfully recover the true spectrum of human judgments and to produce calibrated, inclusive, and robust NLP systems.

1. Taxonomies and Sources of Disagreement

Disagreement arises from a wide array of factors, typically organized in domain-agnostic taxonomies. The major categories include:

• Data Factors: Linguistic ambiguity (polysemy, ellipsis, genre effects), input noise (typos, truncation), epistemic uncertainty (requiring personal or cultural background for interpretation). Ambiguous instances often provoke scattered label choices, indicating no single latent truth (Xu et al., 14 Jan 2026, Jayaweera et al., 20 Jul 2025).
• Task Factors: Annotation prompt phrasing, guidelines, scale granularity, interface artifacts. Poorly specified or complex schema can amplify divergence by allowing multiple valid labeling criteria. Instructional underspecification and schema-induced confusion are sources of systematic disagreement (Jayaweera et al., 20 Jul 2025, Jiang et al., 2022).
• Annotator Factors: Individual identity (moral, cultural values, expertise), group-level demographics (gender, age, nationality), behavioral noise (fatigue, misunderstanding). Systematic disagreements can index genuine perspective splits, while random noise introduces spurious label variation (Xu et al., 4 Aug 2025, Wan et al., 2023).

These categories interact, as ambiguous data is often interpreted differently depending on annotator background or the exact annotation protocol.

2. Label Representations and Data Collection

Disagreement-aware NLP relies on disaggregated label storage and soft-label construction:

• Empirical Distribution: For each instance, the full distribution over annotator votes is retained: $p_c = n_c / \sum_{c'} n_{c'}$, where $n_c$ is the number of annotators who selected class $c$ (Nie et al., 2020, Leonardelli et al., 2023, Leonardelli et al., 9 Oct 2025, Muscato et al., 25 Jun 2025).
• Entropy and Diversity: Entropy $H(P)$ measures the spread of labeling; high entropy signals underlying ambiguity or pluralism (Nie et al., 2020, Xu et al., 2024).
• Annotator and Worldview Metadata: Annotator IDs, demographics (race, gender, age, education), and explicit worldview statements allow construction of conditioning inputs for models, facilitating perspectivist learning and subgroup-specific inferences (Creanga et al., 2024, Wan et al., 2023, Xu et al., 4 Aug 2025).
• Dataset Design: Recent benchmarks (ChaosNLI, LeWiDi, TID-8, POPQUORN) collect dense, repeated judgments (≥5–100 per instance), preserving raw disaggregated annotations and demographic metadata (Nie et al., 2020, Leonardelli et al., 2023, Leonardelli et al., 9 Oct 2025, 2305.14663, Ivey et al., 25 Jul 2025).

3. Modeling Approaches and Architectural Innovations

Multiple approaches implement disagreement-aware modeling at training and inference:

• Soft-label Learning: Models are trained to predict the empirical label distribution via cross-entropy or KL-divergence objectives with targets $P$ (soft labels), rather than one-hot majority targets (Nie et al., 2020, Muscato et al., 25 Jun 2025, Leonardelli et al., 2023, Xu et al., 4 Aug 2025).
• Multi-task Annotator Models: Annotator-specific output heads predict each individual's judgment, with shared encoders for common semantic representation. Predictive uncertainty is derived from output variance (Davani et al., 2021, 2305.14663, Creanga et al., 2024).
• Embedding-based Models: Learned annotator embeddings (and annotation embeddings—Editor's term) are fused with text encodings, allowing instance-specific adaptation and modeling of personal labeling tendencies (2305.14663, Wan et al., 2023, Leonardelli et al., 9 Oct 2025).
• Mixture-of-Experts Architectures: Inputs are routed through specialized sub-networks (experts) keyed by demographic embeddings; DeM-MoE (Demographic-Aware Mixture of Experts) is an example, capturing structured group-level differences (Xu et al., 4 Aug 2025).
• Belief-Level Aggregation and Structured Merging: Disagreement-aware summarization pipelines first extract aspect-polarity belief sets from documents, aggregate conflicting opinions using distance-based merging operators (e.g., Hamming or L1), and convert beliefs to language via prompting (Aghaebe et al., 8 Jan 2026).
• Supervised and Unsupervised Ambiguity Detection: Binary and multi-label classification, hierarchical cascades, and entropy-thresholding methods are used to identify and type ambiguous items, surfacing candidate points for targeted pluralist modeling (Jayaweera et al., 20 Jul 2025).

4. Evaluation Protocols and Metrics

Disagreement-aware evaluation encompasses both population alignment and individual fidelity:

• Distributional Metrics: KL-divergence, Jensen–Shannon Divergence (JSD), cross-entropy, Brier score between predicted and empirical label distributions; lower values indicate closer approximation to human spread (Nie et al., 2020, Muscato et al., 25 Jun 2025, Leonardelli et al., 2023, Leonardelli et al., 9 Oct 2025, Xu et al., 2024).
• Annotator-Level Metrics: Mean Absolute Error (MAE), F1, and Wasserstein distance for per-annotator predictions; per-group MAE for subgroup fairness (Leonardelli et al., 9 Oct 2025, Xu et al., 4 Aug 2025).
• Fairness and Structure Preservation: Descriptive parity gaps (e.g., $\Delta_{\mathrm{acc}}$), relational metrics like Distance-in-Confusion for annotator similarity, and more recently, normative constraints requiring parity in predicted distributions across demographic groups (Xu et al., 14 Jan 2026).
• Calibration and Uncertainty: Expected Calibration Error (ECE), temperature scaling, and distributional cross-entropy are used to gauge model confidence alignment with empirical uncertainty (Xu et al., 2024, Aghaebe et al., 8 Jan 2026).
• Novel Shared Task Metrics: LeWiDi-2025 employs Average Manhattan Distance and Wasserstein distance for population-level tasks, Average Error Rate and Normalized Absolute Distance for perspectivist (per-annotator) prediction (Leonardelli et al., 9 Oct 2025).

5. Practical Applications and Experimental Insights

Disagreement-aware models have demonstrated empirical gains in a wide array of subjective tasks:

• Classification Tasks: Multi-perspective models and soft-label training reliably outperform baseline majority-vote models in macro-F1 and distributional alignment (JSD) across hate speech, abusive language, irony, stance detection, and legal outcome prediction (Muscato et al., 25 Jun 2025, Muscato et al., 2024, Xu et al., 2024).
• Summarization: Belief-level aggregation yields summaries faithfully reflecting conflicting viewpoints, outperforming direct Fusion-of-N approaches especially in smaller LLMs (Aghaebe et al., 8 Jan 2026).
• Legal NLP: Split vote modeling reveals calibration gaps and highlights systematic difficulty in cases with high vote entropy, supporting post-hoc scaling and subgroup-tailored aggregation (Xu et al., 2024).
• Industrial Error Estimation: Ensemble disagreement across multiple model runs is a strong proxy for human error rates, yielding lower mean average error than LLM-based silver labels in keyphrase extraction across multiple languages and domains (Du et al., 2023).
• Simulating Worldviews: Fine-tuned disagreement predictors conditioned on demographics expose stable versus sensitive axes of controversy across simulated annotator pools (Wan et al., 2023).
• Shared Task Findings: LeWiDi-2025 shows that both population-level soft-label prediction and perspectivist (annotator-specific) modeling yield complementary insights; in-context learning (ICL) with annotator examples and metadata embeddings consistently improve performance (Leonardelli et al., 9 Oct 2025).

6. Limitations, Open Challenges, and Future Directions

Disagreement-aware NLP faces a suite of practical and theoretical challenges:

• Annotation Cost and Diversity: Collecting large, richly annotated datasets with demographics and dense repeated labeling is expensive; imbalanced subpopulation size undermines minority opinion recovery (Ivey et al., 25 Jul 2025, Leonardelli et al., 9 Oct 2025).
• Architectural Complexity: MoE models require sufficient data density per subgroup; blending synthetic perspectives demands careful weighting to avoid bias (Xu et al., 4 Aug 2025).
• Interpretability: Disagreement-aware models necessitate novel attribution frameworks to explain divergent outputs across annotators or demographic groups (Xu et al., 14 Jan 2026).
• Normative Fairness: Most existing work is descriptive rather than prescriptive; principled methods for enforcing equity of representation or decision across plural perspectives are underdeveloped (Xu et al., 14 Jan 2026, Wan et al., 2023).
• Generalization and Simulation: Extending beyond binary or categorical labels to multi-class, regression, or reasoning tasks, as well as simulating uncertainty via persona-driven LLM prompting, remains ongoing (Xu et al., 4 Aug 2025, Ni et al., 24 Jun 2025).
• Signal vs. Noise Separation: Bayesian generative models (e.g., NUTMEG) demonstrate that separating systematic disagreement from random noise is both tractable and essential for accurate minority recovery—though they require explicit modeling of subpopulation structure and may not handle intra-group variation (Ivey et al., 25 Jul 2025).

7. Implications for Dataset Curation and Model Deployment

Best practices in disagreement-aware NLP include:

• End-to-end preservation of raw, disaggregated annotations and demographic data; training on soft-labels rather than aggregated majority vote (Nie et al., 2020, Leonardelli et al., 2023, Muscato et al., 25 Jun 2025).
• Use of multiple evaluation metrics capturing both distributional and per-annotator performance; systematic reporting of entropy and uncertainty (Muscato et al., 25 Jun 2025, Leonardelli et al., 9 Oct 2025, Aghaebe et al., 8 Jan 2026).
• Deployment of multi-task architectures, annotator and annotation embeddings, and explicit expert routing for subgroup fairness and calibration (2305.14663, Xu et al., 4 Aug 2025).
• Adaptive annotation workflows, including disagreement prediction via demographics (to budget repeat labeling), and active sampling to ensure minoritized voices are represented (Wan et al., 2023, Creanga et al., 2024).
• Ongoing development of datasets with explicit ambiguity annotation, explainable justifications, and fine-grained metadata for future cross-lingual and cross-domain pluralist modeling (Jayaweera et al., 20 Jul 2025, Leonardelli et al., 9 Oct 2025).

Disagreement-aware NLP thus advances the field toward more robust, calibrated, and ethically responsible language technologies, embracing the diversity of human perspectives as both modeling target and evaluation standard.