SentencePiece Unigram Model

• SentencePiece Unigram model is a probabilistic subword tokenization approach that formulates segmentation as a latent variable problem optimized using EM.
• It jointly optimizes vocabulary and subword probabilities through iterative pruning and dynamic programming, achieving efficient compression for morphologically complex languages.
• The model supports language-independent implementations and robust normalization for non‐Latin scripts, offering improved performance over greedy methods like BPE.

The SentencePiece Unigram model is a probabilistic subword tokenization algorithm central to modern neural text processing. It estimates a probability distribution over candidate subword units and segments input text by maximizing marginal likelihood. Unlike greedy approaches such as Byte-Pair Encoding (BPE), the Unigram model jointly optimizes both the vocabulary and subword probabilities via an Expectation-Maximization (EM) paradigm, often coupled with iterative pruning. This framework yields robust tokenization efficacy in morphologically complex and low-resource languages and forms the foundation for extensible, language-independent tokenization systems such as the open-source SentencePiece library (Kudo et al., 2018, Land et al., 14 Dec 2025).

1. Probabilistic Model Definition

The SentencePiece Unigram model formalizes subword segmentation as a latent variable problem. Let $V=\{s_1,\dots,s_{|V|}\}$ denote the candidate vocabulary of subword pieces, and $S(w)$ the set of all possible segmentations of string $w$ into sequences of pieces from $V$. Each segmentation $S=[s_{i_1},\dots,s_{i_K}]$ concatenates to $w$. The model assigns probabilities $P(s)$ to subwords with normalization $\sum_{s\in V}P(s)=1$.

The probability of a segmentation $S$ is: $P(S)=\prod_{k=1}^K P(s_{i_k})$ The marginal probability of $w$ is obtained by summing over all segmentations: $P(w)=\sum_{S\in S(w)}P(S)$ Training maximizes the corpus log-likelihood: $$L=\sum_{n=1}^N \log P(w_n)=\sum_{n=1}^N \log\sum_{S\in S(w_n)}\prod_{s\in S}P(s)$$ This formulation allows for modeling complex segmentations under uncertainty and is more expressive than greedy, parse-based alternatives (Kudo et al., 2018, Land et al., 14 Dec 2025, Kashirskiy et al., 20 Dec 2025).

2. EM Training and Vocabulary Pruning

Direct maximization of $L$ is intractable due to exponentially many segmentations. SentencePiece employs EM:

• E-step: For each string $w_n$, construct a segmentation lattice where nodes index positions and arcs represent candidate subwords matching segments. Using forward–backward recurrences,
  • $\alpha_n(0)=1$
  • $$\alpha_n(j)=\sum_{s\in V,\ s\text{ ends at }j}\alpha_n(j-|s|)\cdot P(s)$$
  • $\beta_n$ is analogous for the backward pass.
  • Calculate expected counts for each piece $s$ via arc posteriors: $$r_n(s,i) = \frac{\alpha_n(i-|s|)\cdot P(s)\cdot \beta_n(i)}{P(w_n)}$$
  • Summing $r_n(s,i)$ over all $n$ and matching $i$ yields $C(s)$.
• M-step: Update probabilities by normalizing: $P(s)\leftarrow \frac{C(s)}{\sum_{t\in V}C(t)}$
• Pruning: After each M-step (or every few iterations), prune subwords with lowest expected impact on likelihood or smallest expected counts. Pruning can be via likelihood-drop heuristics or simply by top-probability thresholding (Kudo et al., 2018, Land et al., 14 Dec 2025).

Iterative EM and pruning continue until $|V|$ meets the target vocabulary size. Empirical findings indicate that reducing EM sub-iterations, omitting the digamma transform, and applying Final-Style Pruning (FSP) can accelerate training with negligible quality loss (Land et al., 14 Dec 2025).

3. Decoding and Subword Regularization

Given trained subword probabilities, tokenization of new text amounts to finding the most probable segmentation $S^*$: $$S^* = \arg\max_{S \in S(w)} P(S) = \arg\max_{S} \sum_{s \in S}\log P(s)$$ This is efficiently solved via Viterbi dynamic programming on the segmentation lattice, with time complexity $O(M \cdot L)$ for an input of length $M$ and maximum piece length $L$. The model also supports subword regularization: segmentations can be sampled according to the marginal probability $P(S|w)$, optionally with temperature scaling, exposing downstream models to segmentation variability during training (Kudo et al., 2018).

4. Practical Implementation and Parameterization

Key implementation steps:

• Seed Vocabulary Construction: Build $V$ from all substrings up to a threshold length occurring in the corpus. Pretoken-based seed extraction yields superior loss and token compression outcomes compared to suffix-array aggregation (Land et al., 14 Dec 2025).
• Parameters: Empirical tuning of vocabulary size factor ($\gamma$), EM iterations ($k_{EM}$), pruning shrink factor ($\alpha$), early prune threshold, and overshoot ratio impacts convergence, compression, and likelihood minimally for most settings. FSP variant further simplifies training by keeping only the top-$n$ pieces, often matching or exceeding greedy BPE on compression (Land et al., 14 Dec 2025).
• OOV Handling: Ensure that every Unicode character has coverage in $V$ to avoid mapping input to <unk>. Additive smoothing is unnecessary due to the pruning mechanism.
• Integration: The trained model consists of $V$, subword probabilities, and normalization FSTs, all stored in a self-contained format for reproducible deployment (Kudo et al., 2018).

5. Extensions for Non-Latin and Morphologically Rich Languages

Morphologically rich and low-resource languages such as Arabic and Dzongkha benefit from the Unigram model’s global likelihood criterion, which better accommodates complex affixation and inflection than greedy merge-based techniques. Empirical results for Arabic (AraToken) with a comprehensive normalization pipeline yield fertility reduction from 1.35 (BPE) to 1.199 (SentencePiece normalized), and compression gains from 4.60 to 5.03 chars/token (Kashirskiy et al., 20 Dec 2025). The normalization pipeline addresses Unicode decomposition, orthographic unification (e.g., Alif variants), numeral and punctuation mapping, tatweel removal, and diacritic handling. For Dzongkha, SentencePiece achieves optimal subword fertility (0.79), minimal proportion of continued words (0.09), and superior normalization compared to WordPiece and BPE (Wangchuk et al., 18 Sep 2025).

The Language Extension Pipeline (LEP) integrates new vocabularies into existing models (Qwen3-0.6B), leveraging mean subtoken embedding initialization, gradient masking for old embeddings, and selective unfreezing of transformer layers. This enables efficient adaptation to new scripts with under 0.01% of pretraining cost, demonstrated by evaluation loss reduction from 8.28 to 2.43 after 800 steps on Arabic text (Kashirskiy et al., 20 Dec 2025).

6. Comparative Evaluation and Metrics

Performance assessment utilizes metrics tailored for subword quality:

Metric	Ideal Value	Definition
Fertility	1.0	tokens/word; lower indicates better compression
Compression	high	chars/token; higher is better
Proportion of Continued Words	0.0	fraction of words split into multiple tokens; lower implies less fragmentation
Normalized Sequence Length	—	sequence length vs. baseline; lower is more efficient
Execution Time	—	wall-clock runtime per input loop

SentencePiece Unigram frequently achieves lowest fertility and highest compression in comparative studies. For Arabic with diacritic dropping, SentencePiece normalized yields fertility 1.199 and compression 5.03, outperforming BPE and WordPiece (Kashirskiy et al., 20 Dec 2025). For Dzongkha, SentencePiece achieves fertility 0.79, continued word proportion 0.09, and fastest inference time (131 ms/loop) (Wangchuk et al., 18 Sep 2025). Trade-offs between compression and likelihood (e.g., FSP vs. standard pruning) are well characterized: FSP may yield 1.4% fewer tokens at 0.5–1.5% higher loss out-of-domain (Land et al., 14 Dec 2025).

7. Implications and Limitations

The SentencePiece Unigram model provides a language-independent framework for subword tokenization with effective modeling of morphological phenomena and extensibility to non-Latin scripts. Its probabilistic EM-based inference avoids the over-fragmentation seen in greedy approaches and directly optimizes corpus likelihood. Language-specific normalization, targeted vocabulary pruning, and efficient integration strategies underpin superior performance in morphologically rich and low-resource contexts. Practical simplifications (e.g., FSP) yield substantial implementation efficiency without sacrificing quality.

A plausible implication is that the Unigram model's principled segmentation and robust pruning make it especially suitable for rapid extension of LLMs to new scripts or resource-scarce settings. However, its computational cost in training (especially EM iterations and likelihood calculations) may exceed that of BPE, though tokenization at inference remains efficient. Overall, the Unigram model is technically preferred where morphological alignment and compression are critical (Land et al., 14 Dec 2025, Kashirskiy et al., 20 Dec 2025, Wangchuk et al., 18 Sep 2025, Kudo et al., 2018).