Distant-Supervision Data Construction

• Distant-supervision data construction is a methodology that automatically generates weakly-labeled corpora by aligning raw text with external resources like knowledge bases and gazetteers.
• It employs precise string-matching and rule-based heuristics for label assignment, facilitating scalable NER and relation extraction even in low-resource languages.
• Empirical results show that combining noisy distant supervision with minimal gold data significantly improves model performance through effective noise-handling techniques.

Distant-supervision data construction refers to a set of methodologies for automatically generating annotated corpora in the absence of large-scale gold-standard labels, relying instead on external resources such as knowledge bases, gazetteers, or heuristic alignment rules. The appeal of distant supervision lies in its capacity to rapidly expand training datasets for tasks like named entity recognition (NER), relation extraction, and event temporal order prediction—especially for low-resource languages or domains—while introducing specific forms of annotation noise that necessitate robust noise-handling strategies.

1. Foundations of Distant-Supervision Data Construction

The core distant-supervision paradigm extracts weak labels by aligning unlabeled raw text with external supervision sources. In the NER context for Hausa and Yorùbá, named entity lists were constructed from Wikipedia, Wikidata, commercial name dictionaries, and curated personal-name lexica. Each potential token in the target textual corpus was normalized (lowercased, diacritic removed) and string-matched to these gazetteers; exact matches governed entity labeling (such as "B-PER" for people). In the Yorùbá setup, native speakers provided minimal keyword sets for date/time detection, which together with numeral pattern rules allowed for basic DATE entity annotation.

All list elements were subject to post-filtering based on minimal character length constraints (e.g., requiring entity names ≥2 or ≥3 characters depending on entry origin). Matching was strictly string-based, with no fuzzy or embedding-based approximate matching involved.

2. Annotation and Label Assignment Heuristics

The assignment of weak labels is governed by precise string-matching or rule-based heuristics. The BIO scheme is used for span labeling, with multiword matches yielding "I-" continuation labels following "B-". For special classes (e.g., DATE in Yorùbá), rule-based criteria incorporate both position (token immediately after a detected date keyword) and pattern (matching date digit regex). There is no in-corpus disambiguation mechanism; overlapping matches are optionally resolved by preferring the longer gazetteer entry.

These procedures are formalized in the data as:

• For each token $t$ in a sentence, set $s[i].\text{label} = \text{PER}$, $\text{LOC}$, $\text{ORG}$, $\text{DATE}$, or $\text{O}$ based on normalized membership in the respective resource lists or keyword sets.

3. Pipeline for Constructing Noisy Labeled Corpora

The standard pipeline comprises:

1. Corpus collection: Assemble unlabeled running text (e.g., ~10k Hausa news/forum sentences; ~20k Yorùbá tokens from Global Voices).
2. Normalization and tokenization: Lowercase text, remove diacritics, separate on whitespace and punctuation.
3. Heuristic labeling: Apply the string-match and pattern rules for each token, yielding a noisy BIO-labeled dataset.
4. Filtering: Discard matches failing the minimal-length criteria and, optionally, resolve overlaps in favor of longer entries.
5. Corpus aggregation: Merge the annotated sentences into a single "noisy" training set ready for downstream modeling.

No probabilistic or threshold-based filtering is applied beyond the empirically set length constraints.

4. Noise-Handling and Model Adaptation Techniques

Downstream models trained on such noisy corpora require explicit noise-handling mechanisms to approach the performance of models trained on hand-labeled data. The principal denoising approaches utilized include:

• Noise-channel/EM modeling: Introduces a noise transition matrix $T$ where $T_{ij} = P(\text{noisy}=y_j \mid \text{clean}=y_i)$, maximizing the expected noisy-label likelihood via expectation-maximization jointly over model parameters $\theta$ and $T$.
• Confusion-matrix corrections: When a small seed of gold data is available, estimate $T$ using empirical counts, back-project noisy predictions $p_\theta(y \mid x)$ to the clean label space.
• Cleaning networks: A learned neural mapping from input representations $h(x)$ to "cleaned" logits, with final predictions $p_\theta(f_\phi(h(x)))$.

In all cases, cross-entropy loss is computed between the possibly denoised predictions and observed noisy labels.

5. Empirical Performance and Dataset Statistics

Quantitative analysis reveals:

Language	Tokens/Sentences	Non-O tokens (approx.)	Person F1 (train)	Key Precision/Recall (held-out)
Hausa	10,000 sents	Only PERSON	N/A	N/A
Yorùbá	19,559 tokens	c. 1,900	33–46% (by list)	PER: R≈25%, P≈50%; LOC: R≈54%, P≈73%

Supplementing Wikidata gazetteers with commercial and hand-curated lists significantly boosted recall, with overall F1 for PERSON rising from 33% to 46%. The date detection heuristics yielded DATE recall ≈ 66% and precision ≈ 37%.

In extremely low-resource settings (e.g., 1,000–2,000 gold tokens), augmenting with noisy distant-supervision data increased Bi-LSTM F1 from ≈9% to ≈36%. Even with BERT-based NER, combining 1,000 gold tokens with noisy data raised F1 from ≈34% to ≈47%.

Noise-handling via confusion-matrix approaches recouped an additional ≈2 F1 points relative to naive noisy-label training, with overall distant-supervision NER baselines achieving 40–50% F1—often surpassing the performance of gold-only models at the lowest resource constraints.

6. Implications and Practical Guidance

These findings demonstrate the effectiveness of explicit, resource-driven distant-supervision in generating NER training data for low-resource languages, where full annotation is infeasible. Simple list-based heuristics—augmented with minimal native-speaker pattern rules—can yield a viable baseline for downstream sequence tagging tasks. Noise arises primarily from gazetteer incompleteness, ambiguous entity matches, and limitations of rule-based tagging, but standard statistical noise-adaptation methods mitigate their adverse effects.

Notably, model overfitting to noise becomes apparent as additional gold data is added (beyond ≈5,000 tokens), highlighting the necessity for careful calibration of distant and gold supervision ratios.

A plausible implication is that in other low-resource domains, similar gazetteer- and pattern-driven pipelines, combined with noise-adaptation layers, may offer a scalable route to bootstrap task-appropriate labeled corpora without the need for large-scale hand annotation.