District Guided Tokens (DGT)

• DGT is a model-guidance technique that prepends a district token to input sequences, enabling transformer models to condition IPA transcription on region-specific phonological features.
• The technique utilizes explicit district identifiers within models like ByT5 and mT5 to reduce dialectal confusion and significantly lower word error rates, especially in high OOV contexts.
• DGT has broad applicability for low-resource, multi-dialect languages, offering scalable improvements in tasks such as machine translation, speech synthesis, and dialect-aware named-entity recognition.

District Guided Tokens (DGT) are a model-guidance technique deployed in sequence-to-sequence (seq2seq) natural language processing systems to address the challenge of conditional text–to–International Phonetic Alphabet (IPA) transcription in languages exhibiting pronounced regional dialect variation. The method introduces explicit district-level information by prepending a dedicated district token to each input sequence, enabling transformer-based models to condition their output (e.g., IPA transcription) on district-specific phonological features. Originally developed to support Bengali text transcription across six linguistically diverse districts of Bangladesh, DGT demonstrates substantial improvements in transcription quality, especially in regions lacking standardized spelling conventions or characterized by significant out-of-vocabulary (OOV) lexical diversity (Islam et al., 2024).

1. Formulation and Core Mechanism

DGT reframes dialectal transcription as a conditional seq2seq problem. Let the canonical input be represented as $x = (x_1, x_2, \dots, x_n)$, where $x_i$ is either a byte (for ByT5) or a subword unit (for other models). A district identifier $d \in \{d_1, d_2, \dots, d_6\}$, corresponding to Dhaka, Chittagong, Khulna, Sylhet, Barisal, and Rajshahi, is encoded via a special token $[D_d]$ prepended to $x$. The augmented input is $x' = ([D_d], x_1, x_2, ..., x_n)$. The learning objective is to accurately model the conditional probability of an IPA sequence $y = (y_1, ..., y_T)$ given $x'$, with the autoregressive factorization:

$$P(y \mid x') = \prod_{t=1}^T P(y_t \mid y_{<t}, x')\;.$$

This approach enables the model to learn distinct mappings and phonological rules for each regional variant through the explicit district conditioning of the input.

2. Model Architectures and Training Paradigm

DGT has been implemented within several transformer encoder–decoder architectures, primarily ByT5 (tokenizing at the byte level), but also mT5 (multilingual SentencePiece), BanglaT5 (Bangla-specific subword), and umT5 (Unimax multilingual). Model parameters $\theta$ encompass attention weights, feed-forward layers, and token embeddings, expanded to accommodate the new district tokens. Training minimizes the standard cross-entropy loss over $(x', y)$ pairs:

$$\mathcal{L}(\theta) = -\sum_{t=1}^T \log P_\theta(y_t \mid y_{<t}, x')$$

By updating district-specific embeddings during fine-tuning, the model internalizes district-conditioned phonological patterns. Only a single fine-tuning run per architecture is required, negating the need for district-specialized models.

3. Dataset Construction and Preprocessing

The technique leverages the “IPA Transcription of Bengali Texts” corpus and the Bhashamul Kaggle challenge dataset. This encompasses texts from six districts, with a split of 80% training and 20% hidden test (IPA sequences withheld for evaluation). Within the training portion, a further 90%/10% split yields approximately 23,100 training examples and 2,600 dev examples; the hidden test set contains 8,941 examples. Average text length is 31.9 characters (max 306), while IPA transcriptions average 38.1 symbols (max 350). Of the 10,487 unique words in the test set, 4,926 are OOV relative to training (≈ 47%). Preprocessing includes normalization of Unicode (NFC), manual preservation of dialect-aware spelling, and appending the district token as a unique vocabulary item per tokenizer.

4. Experimental Setup and Hyperparameter Regime

Four model/tokenization configurations were evaluated:

• ByT5 (byte-level)
• mT5 (multilingual SentencePiece)
• BanglaT5 (Bangla-only subword)
• umT5 (Unimax multilingual)

Common hyperparameters across experiments include: AdamW optimizer, learning rate $3\times 10^{-4}$, weight decay $1\times 10^{-2}$, batch size 4, 10 epochs (with early stopping on dev), and maximum input/output length of 1,024 tokens. Hardware setup involved dual NVIDIA Tesla T4 GPUs (15 GB per card).

5. Empirical Results and Analysis

Performance was evaluated using Word Error Rate (WER):

$\mathrm{WER} = \frac{S + D + I}{N} \times 100\%\,,$

where $S$ is substitutions, $D$ is deletions, $I$ is insertions, and $N$ is the reference word count. DGT ablation studies reveal robust improvements across all model types. Table 1 summarizes aggregate WER metrics:

Model	WER (no DGT)	WER (DGT)	Absolute Improvement (pp)
ByT5	3.00%	2.07%	–0.93
umT5	4.70%	3.96%	–0.74
BanglaT5	62.00%	61.09%	–0.91
mT5	30.50%	28.19%	–2.31

District-wise, ByT5 with DGT yields average WER reductions from 0.55 pp (Dhaka) to 0.90 pp (Sylhet). Notably, ByT5 consistently outperforms word/subword models, attributed to its byte-level OOV robustness. DGT delivers another layer of improvement by reducing dialectal confusion, especially in districts with high non-standard spelling.

6. Applicability and Broader Significance

DGT’s principal impact lies in enabling regionally conditioned NLP systems for low-resource, multi-dialect languages. With district tokens functioning as pseudo-language tags, models can learn and apply distinct phonological and orthographic adaptations without fragmenting the training corpus or proliferating model variants. This paradigm is extensible beyond phonetic transcription; plausible implications include its utility in machine translation, speech synthesis, and multi-dialect named-entity recognition. The explicit encoding of dialectal context via guide tokens addresses regional diversity in linguistic phenomena and scales efficiently with modest region-annotated datasets.

7. Interpretations and Prospective Developments

The DGT method elucidates the role of explicit regional context in addressing the systemic limitations of standard tokenization and modeling regimes in dialectally diverse languages. Key findings indicate that byte-level encoding, coupled with district guidance, is effective in circumstances of high OOV prevalence and orthographic non-standardization. This suggests that similar tag-based conditioning is likely to generalize to other under-represented languages presenting heterogeneous dialectal features. As NLP systems increasingly address granular linguistic diversity, district or region-guided token strategies represent a simple and scalable lever to improve sequence transduction fidelity (Islam et al., 2024).