Corpus Clarification Framework

• Corpus clarification is a preprocessing framework that restructures heterogeneous texts into self-contained argumentative units by segmenting, labeling, and rewriting content.
• It employs techniques such as LLM-based segmentation, argumentative structure detection, and prompt-driven rewriting with metrics like BERTScore and ROUGE-L for quality assurance.
• The approach has been validated on large-scale citizen consultation datasets, demonstrating improved thematic cohesion and enhanced performance in opinion mining and clustering.

Corpus clarification denotes a systematic preprocessing framework that restructures noisy, multi-topic, or pragmatically fragmented textual contributions into mono-topic, self-contained argumentative or analytic units optimized for downstream computational analysis. Originally introduced to facilitate principled analysis of democratic citizen consultations, the concept operationalizes three tightly-coupled steps: segmentation (argumentative unit extraction), labeling (argumentative structure detection), and rewriting (unit clarification). Corpus clarification explicitly formalizes the means by which heterogeneous, multi-idea, and context-dependent texts—common in real-world participatory settings—are rendered suitable for information extraction, opinion mining, topic modeling, and clustering at scale. The term also encompasses metrics and infrastructure for benchmarking preprocessing quality and downstream utility (Lequeu et al., 21 Jan 2026).

1. Formal Definition and Conceptual Scope

Let $D = (t_1, t_2, ..., t_n)$ be the token sequence of a raw contribution in a large-scale consultation corpus. Corpus clarification is the composite transformation

$f: D \longrightarrow \{u'_1, ..., u'_k\}$

where each $u'_i$ is a clarified argumentative unit. The pipeline factorizes $f$ as:

• $f_1$: Segment — extraction of argumentative units ($u_1, ..., u_m$).
• $f_2$: Classify — detection of argumentative structure for each $u_i$; labels $c_{ij} \in \{$Statement, Solution, Premise$\}$.
• $f_3$: Rewrite — clarification/normalization of $u_i$, producing $u'_i$ requiring mono-topic focus, self-containment, and stylistic normalization.

The complete mapping is: $\{u'_i\}_{i=1}^m = (f_3 \circ f_2 \circ f_1)(D)$ with the properties:

• Mono-topic: each $u'_i$ addresses a coherent, atomic idea.
• Self-contained: $u'_i$ encodes all requisite contextual information from $D$.
• Stylistically normalized: consistent register and explicitness (Lequeu et al., 21 Jan 2026).

2. Framework Architecture and Algorithmic Steps

Segmentation (Argumentative Unit Extraction)

Segmentation identifies the minimal spans in $D$ each covering a unique topic or claim:

• Prompted LLMs or fine-tuned small LLMs output a bullet list of spans.
• Evaluation uses WindowDiff:

$$\mathrm{WindowDiff}(A,B) = \frac{1}{N - k} \sum_{i=1}^{N-k} \bigl|b_A(i, i+k) - b_B(i, i+k)\bigr|$$

where $b_A(i, i+k)$ counts boundaries in the window $i$ to $i+k$ (Lequeu et al., 21 Jan 2026).

Argumentative Structure Detection

Each extracted unit $u_i$ is then labeled:

• Span or token-level classifier assigns labels Statement, Solution, Premise.
• Models: LLM in few-shot prompt mode, or encoder-based span taggers.
• Loss:

$$\mathcal{L}_{AS} = -\sum_{i, t} \sum_{c} y_{i,t,c} \log P(c \mid u_i, t)$$

Clarification and Rewriting

Finally, each $u_i$, together with its detected structure, is rewritten into $u'_i$:

• Prompt-based rewriting with context $D$ ensures $u'_i$ is self-sufficient.
• Evaluation by overlap metrics: BERTScore, ROUGE-L.

3. Data Resources and Annotation Paradigm

The “GDN-CC” dataset operationalizes corpus clarification for French citizen consultations:

• Source: 355,000 contributions from the Grand Débat National platform (2019).
• Annotation pool: 1,231 contributions $\rightarrow$ 2,285 argumentative units (AUs).
• Types: 21.2% Statement, 57.9% Solution, 20.9% Premise.
• Thematic coverage: balanced across four policy domains.

Inter-annotator agreement:

• AU segmentation: WindowDiff = 0.09 (mean), token-overlap micro-F1 = 0.72.
• Structure detection: overall agreement 0.65 (class-wise: Solution 0.79, Premise 0.59, Statement 0.50), with increased agreement (0.743) when merging Premise+Statement.
• Clarification quality: evaluated against human rewrites using string and BERTScore metrics (Lequeu et al., 21 Jan 2026).

4. Model Architectures, Training, and Evaluation Metrics

Corpus clarification is architecturally modular:

• AU Extraction: LLM (e.g., GPT-4.1), small LLMs (e.g., Qwen2.5-7B, Gemma-2-9B) using prompt-based outputs or gradient descent on cross-entropy loss.
• Structure Detection: Sequence or span models with softmax output over AU segment classes, optimized with cross-entropy loss.
• Clarification: Prompted rewriting with context inclusion; decoded outputs are measured with BERTScore and ROUGE-L against human-annotated gold clarifications.

Performance metrics: $$\mathrm{Precision} = \frac{TP}{TP+FP},\quad \mathrm{Recall} = \frac{TP}{TP+FN},\quad F_1 = 2\frac{\mathrm{Precision} \times \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}$$ Clustering and downstream analysis use ARI (Adjusted Rand Index) and Silhouette Score.

Results (best SLM vs. GPT-4.1):

• AU extraction: Micro-F1 0.75.
• Structure detection: Micro-F1 up to 0.76.
• AU clarification: BERTScore up to 0.86, ROUGE-L up to 0.60. Open-weight SLMs match or surpass proprietary LLMs for all tasks (Lequeu et al., 21 Jan 2026).

5. Downstream Applications: Impact on Opinion Clustering and Analysis

Corpus clarification directly enhances thematic cohesion, facilitates topic modeling, and supports auditable citizen opinion summarization:

• Pipeline: SentenceTransformer embedding, UMAP reduction, HDBSCAN clustering.
• Evaluation by pairwise “LLM-as-Judge” (pseudo GPT-4-nano): clarified units yield higher within-cluster coherence than raw or monosegmented inputs; clarified clusters judged preferable in 91%+ of pairings.
• Practical consequence: opinion and policy extraction operates on well-defined, discrete atomic units rather than underdetermined or ambiguous fragments (Lequeu et al., 21 Jan 2026).

Potential applications:

• Political science: extraction of proposals, argument mining in participatory platforms.
• Topic modeling: improved interpretability due to mono-topic segmentation.
• Sentiment analysis: robust normalization reduces socio-stylistic confounds.
• Platform design: enables real-time standardization and analysis pipelines.

6. Scaling, Automation, and Open Data Release

A fully automatic extension “GDN-CC-large” contains 300,748 clarified argumentative units from 240,000 contributions. Distribution balance: 31.2% Statements, 56.4% Solutions, 12.4% Premises. This resource constitutes the largest annotated citizen consultation corpus to date (Lequeu et al., 21 Jan 2026).

Release on the French governmental open-data platform ensures reproducibility, transparency, and ethical auditability for democratic deliberation studies. All preprocessing code, prompt templates, and benchmarking scripts are part of the data release, enabling direct extension to other languages and contexts.

7. Limitations, Open Challenges, and Extensions

• Ambiguity in segment boundaries and structure labeling persists; agreement plateaus at macro-F1 ≈ 0.77 even for expert annotators.
• Nontrivial variance in annotation spans; human annotators sometimes merge Premise and Statement.
• Rewriting is bottlenecked by LLM prompt design and domain transferability.
• Some loss of nuanced context may occur when forcibly segmenting discourse; applications requiring cross-unit pragmatic inference should model such global dependencies.
• The framework is designed for participatory and consultation genres; adaptation to other domains (e.g., scientific or technical corpora) may require task-specific argumentation schemas.

Corpus clarification has catalyzed refined methodologies for preprocessing, analysis, and interpretation of large-scale, multi-topic democratic text collections and establishes new empirical baselines for argument extraction quality at internet scale (Lequeu et al., 21 Jan 2026).