Byte-Level Byte-Pair Encoding

• Byte-level BPE is a tokenization algorithm that starts with bytes or Unicode code points and iteratively merges frequent pairs to build a fixed vocabulary.
• It achieves efficient compression and tokenization parity, though complex scripts often require grapheme-level improvements for accurate representation.
• Empirical results show that using grapheme extraction (GPE) slightly outperforms vanilla methods by improving compression ratios and sequence consistency in abugida languages.

Byte-level Byte-Pair Encoding (BPE) is a tokenization algorithm that initializes its vocabulary with byte-level units or Unicode code points and iteratively merges the most frequent adjacent pairs to construct increasingly larger tokens until a fixed vocabulary size is reached. Within this family of methods, recent research has highlighted fundamental limitations for complex script languages, motivating the introduction of enhancements such as Grapheme Pair Encoding (GPE) (Velayuthan et al., 2024). Both BPE and GPE share the same core merge procedure, differing solely in the atomic units (bytes/code points versus Unicode grapheme clusters) from which their vocabularies are grown, a distinction of particular importance for scripts where characters may span multiple bytes or code points.

1. Formal Definition and Algorithmic Workflow

Standard Byte-Pair Encoding treats either UTF-8 bytes or UTF-32 code points as atomic units. The starting vocabulary $V_0$ consists of all such units observed in the training corpus. The algorithm repeatedly merges the most frequent adjacent pair of units, expanding the vocabulary iteratively until the specified size ($V_{\text{size}}$) is reached. At each step, corpus segments are re-tokenized to replace merged sequences.

Grapheme Pair Encoding (GPE) generalizes this by initializing $V_0$ with Unicode grapheme clusters, extracted using a compliant grapheme segmentation library. The downstream process, including merge counting, frequency computation, and vocabulary growth, is identical to BPE.

The pseudocode for GPE, abstracted from Algorithm 1 in the source, is as follows:

!!!^^^^1வணக்கம்1^^^^!!!

All other steps and tie-breaking heuristics are inherited from canonical BPE (Velayuthan et al., 2024).

2. Pre-tokenization and Grapheme Extraction Pipeline

The pre-tokenization pipeline enforces a language- and script-agnostic segmentation strategy. It proceeds as follows:

1. Raw Text Segmentation: The corpus is segmented into pre-tokens via a regular expression (e.g., whitespace, punctuation).
2. Grapheme Cluster Extraction: Each pre-token is further processed into Unicode grapheme clusters. For abugida scripts (e.g., Tamil, Sinhala, Hindi), this operation is crucial to preserve atomic units such as consonant-vowel combinations or joined forms.
3. Grapheme Sequences: Each resulting grapheme serves as an atomic symbol for merges in GPE.
4. Corpus Conversion: The full collection of grapheme sequences forms the substrate for BPE-style merge operations.

For Tamil word “வணக்கம்!”, regex pre-tokenization yields ["வணக்கம்", "!"]. The grapheme cluster extraction on “வணக்கம்” produces ["வ", "ண", "க்", "க", "ம்"]. Each cluster is included as an initial vocabulary element (Velayuthan et al., 2024).

3. Comparative Analysis: Byte-Level vs. Grapheme-Level Tokenization

Theoretical Characteristics

• Linguistic Integrity: GPE aligns atomic units with human-perceived characters, unlike byte-level BPE, which is agnostic to script boundaries and may fragment meaningful script features (e.g., Devanagari matras, Tamil pulli).
• Vocabulary Efficiency: GPE avoids redundant tokens for spurious or ambiguous code point combinations; byte-level approaches produce wasteful vocabularies for scripts with many grapheme combinations.
• Compression and Parity: GPE enables better compression and more equitable sequence lengths across languages.

Trade-offs

• Grapheme Segmentation Overhead: GPE introduces a small, one-time computational cost for extracting grapheme clusters.
• Dependency on Segmentation Quality: GPE’s effectiveness depends on the Unicode compliance and accuracy of the external grapheme segmentation library.
• Implementation Complexity: While the merge routine remains unchanged, the integration of grapheme extraction minimally increases system complexity.

4. Evaluation Metrics and Experimental Protocol

The intrinsic evaluation relies on two main ratios:

• Compression Ratio ($\mathrm{CR}$): $$\mathrm{CR} = (\mathrm{Original\,sequence\,length})/(\mathrm{Tokenized\,sequence\,length})$$; higher values indicate better compression.
• Tokenization Parity ($\mathrm{TP}$): $\mathrm{TP}(A{\to}B) = |t(A)| / |t(B)|$, where $t(\cdot)$ is the token sequence output; $\mathrm{TP} \sim 1$ indicates parity.

Setup Summary

• Languages: Tamil, Sinhala, Hindi.
• Training Data: 150k randomly-sampled Tamil sentences (Samanantar corpus).
• Testing Data: FLORES+ dev-test sets for all target languages.
• Tokenization Baselines: GPT-2/4, Llama 3, FLAN-T5, Gemma 2, Aya, mBERT, mT5, mBART, NLLB; byte-level: ByT5 (UTF-8), CANINE (UTF-32); vanilla: BPE, Unigram, WordPiece; grapheme-based: GPE.
• Fixed Vocabulary Size: 5,000.
• Intrinsic Metrics: $\mathrm{CR}_{\max}$ and $\mathrm{TP}_{\min}$ (Velayuthan et al., 2024).

5. Empirical Results and Quantitative Comparison

Byte-Level Baselines vs. Grapheme Extraction

Model	$\mathrm{CR}_{\max}$, Tamil	$\mathrm{TP}_{\min}$, Tamil
CANINE (UTF-32)	0.98–0.99	1.17
ByT5 (UTF-8)	0.37	3.20
Grapheme-based (GPE)	1.55	0.76

Key findings:

• CANINE nearly matches English in both compression and parity, but at the cost of a large implicit vocabulary.
• ByT5 (byte-level UTF-8) suffers severe sequence inflation for abugida scripts (sequence lengths tripled).
• GPE achieves ~1.4–1.6× better compression and brings tokenization parity close to or better than English benchmarks (Velayuthan et al., 2024).

GPE vs. Vanilla Subword Algorithms

On Tamil with 150k sentences and 5,000 vocab:

Tokenizer	Compression Ratio (CR)
BPE	4.32
Unigram	4.31
WordPiece	4.12
GPE	4.36

GPE slightly outperforms all vanilla tokenization methods, confirming a modest but consistent gain from grapheme-level initialization (Velayuthan et al., 2024).

6. Ablation Study and Impact of Pre-tokenization

The ablation diagnoses the relative impact of pre-tokenization quality versus algorithmic enhancements:

• (A) BPE with suboptimal (GPT-2 regex) pre-tokenizer: $\mathrm{CR} \approx 1.36$
• (B) BPE with whitespace pre-tokenization: $\mathrm{CR} \approx 4.31$–$4.32$
• (C) GPE with whitespace pre-tokenization: $\mathrm{CR} \approx 4.36$

These findings establish that pre-tokenizer design dominates improvements, while initializing with graphemes (GPE) adds a further incremental gain over a robust pre-tokenizer baseline (Velayuthan et al., 2024).

7. Limitations and Prospects for Future Investigation

Limitations

• The reported results are limited to intrinsic metrics (compression and parity); no evaluation on downstream tasks (e.g., translation, language modeling perplexity) has yet been performed.
• Only the abugida languages Tamil, Sinhala, and Hindi have been studied. Other scripts (e.g., abjad, logographic) may necessitate further adaptation.
• GPE is implemented only atop BPE, not alternative subword learners (Unigram, WordPiece).
• Static vocabulary size (5,000) may not represent cross-language optimality.

Directions for Future Work

• Extending grapheme-based initialization to Unigram and WordPiece tokenizers.
• Adapting grapheme extraction routines for a broader set of scripts, including abjad and logographic writing systems.
• Evaluating impact on extrinsic tasks such as LM perplexity, BLEU score for machine translation, and generative output quality.
• Examining dynamic vocabulary scaling methods adapted to grapheme statistics.
• Analyzing computational efficiency, training speed, and inference memory under grapheme-based tokenization regimes (Velayuthan et al., 2024).

---

For full methodology, ablation protocol, and language-specific findings, see the comprehensive discussion and supplementary sections of "Egalitarian Language Representation in LLMs: It All Begins with Tokenizers" (Velayuthan et al., 2024).