Lexically Grounded Subword Segmentation

• Lexically grounded subword segmentation is a technique that decomposes words into meaningful morphemes by leveraging explicit lexical knowledge to respect true morphological boundaries.
• It integrates dictionary-driven, rule-based, and hybrid methods to improve segmentation precision, reduce token fertility, and enhance downstream task performance such as MT, ASR, and POS tagging.
• Empirical results show improved handling of rare words, reduced perplexity, and increased accuracy in low-resource and morphologically complex languages, supporting more efficient NLP pipelines.

Lexically grounded subword segmentation is the process of decomposing words into linguistically meaningful subunits—such as morphemes or inflected forms—guided by explicit lexical knowledge rather than purely statistical co-occurrence or unsupervised heuristics. This methodology stands in contrast to frequency-driven, agnostic segmenters such as BPE or unigram LM models, and is designed to respect true morphological boundaries and lexical semantics across typologically diverse languages, including highly inflected and agglutinative languages.

1. Theoretical Foundations and Motivations

The principal goal of lexically grounded subword segmentation is to address the inadequacy of frequency-based subword vocabularies, particularly in morphologically rich languages where affixes and stem alternations are crucial for meaning. In such settings, standard unsupervised segmenters often over-split (fragmenting meaningful morphemes) or under-split (failing to reveal structure), impeding downstream representation learning and OOV coverage. Lexical grounding ensures each subword corresponds to a genuine morpheme or lexical unit, yielding tokens that are semantically interpretable, enhance parameter efficiency, and better support typologically diverse languages (Brahma et al., 14 Apr 2025, 2207.13333, Libovický et al., 2024, Nzeyimana et al., 4 Jul 2025).

Empirical analyses confirm significant reductions in token “fertility”—the number of subword pieces per word—and marked improvements in segmentation precision, rare-word representation, and task-specific accuracy for POS tagging, language modeling, NMT, ASR, and retrieval (Libovický et al., 2024, Brahma et al., 14 Apr 2025, 2207.13333, Nzeyimana et al., 4 Jul 2025, El-Kishky et al., 2019).

2. Lexically Grounded Segmentation Algorithms

Dictionary-Driven and Rule-Based Approaches

Manually constructed dictionaries of stems, affixes, and derivational relationships provide the most direct lexical grounding. For example, Czech segmentation employs a derivational network (DeriNet) and an inflectional lexicon (MorfFlex) to build a mixed derivational/inflectional graph. Segmentation proceeds by aligning stems via longest common substrings across derivational edges and propagating boundary information, yielding deterministic splits that coincide with known morphological boundaries (Macháček et al., 2018).

In Tamil and Kannada ASR, weighted FSTs are constructed from curated prefix/infix/suffix inventories for each word category. These subword grammar WFSTs accept only sequences corresponding to legitimate morphological patterns. Out-of-vocabulary forms are addressed using a universal fallback WFST looped over atomic subwords and characters, with cost-based selection (2207.13333).

Morphology-Aware Pre-Tokenization for Statistical Segmenters

For scalability, pre-tokenization by linguistically motivated segmenters such as Morfessor (Grönroos et al., 2020, Libovický et al., 2024) or supervised neural sequence models (Brahma et al., 14 Apr 2025) is integrated upstream of BPE or Unigram LM. The resulting morph-like units bootstrap subsequent merge operations, anchoring data-driven vocabulary construction to boundaries with high morphological plausibility.

In Hindi and Marathi, a hybrid of lookup-based morpheme segmentation, model-driven expansion, and sandhi handling yields a pre-split text that is further segmented and constrained by “Constrained BPE” (CBPE), which incorporates script-dependent constraints such as dependent-vowel attachment (Brahma et al., 14 Apr 2025).

Approach	Lexical Knowledge Source	Handling OOV / Coverage
DeriNet/MorfFlex	Human-curated dictionaries	Defaults to no split for missing
SG-WFST (ASR)	Manually listed subwords	U-WFST fallback to atomic chars
MorphMine	Distributional statistics	Only supports substrings in inventory
MorphTok/CBPE	Look-up/model+rules + BPE	Model expansion for OOVs

3. Unsupervised and Probabilistic Lexically Grounded Models

Bayesian and Information-Theoretic Models

Morfessor EM+Prune casts segmentation as joint inference over a lexicon of morphs and their probabilities, optimizing a penalized likelihood objective (MDL prior). EM estimates segmentation posteriors, while lexicon pruning removes morphs not justifiable by the corpus likelihood and prior cost. Each retained subword is a learned lexical entry, and only forms with recurrent lexical support are allowed (Grönroos et al., 2020). MorphMine applies a parsimony principle, dynamically learning hierarchical segmentations to maximize coverage using the fewest units, with tie-breaking based on substring frequency, and global consistency enforced by multi-pass resegmentation and substring frequency pruning (El-Kishky et al., 2019).

Neural Segmental Models

The SNLM framework encodes a generative semi-Markov process over segments, with segment probability distributed between a character-level LSTM generator and a learned lexicon memory. The model is regularized for expected segment length and optionally grounded in multimodal input (e.g., visual context). The lexicon memory aligns the induced lexical units with frequent substrings, and inference proceeds by dynamic programming over segmentations (Kawakami et al., 2018).

4. Embedding and Contextualization with Lexically Grounded Units

Lexically grounded segmentation enables direct integration of morpheme-based tokens into representation learning. In FastText variants, each word representation sums over learned embeddings of its morphemes, as discovered by parsimonious segmentation (El-Kishky et al., 2019).

“LexICAL” style approaches learn subword embeddings algebraically: by mapping each candidate subword into a word embedding space, via co-occurrence statistics and matrix factorization (closed-form pseudo-inverse). Given these embeddings, segmentation of a word is scored for semantic coherence (via the cosine similarity between the word and subword vectors), with a length-penalty regularizer. Segmentation is optimized per word by dynamic programming, and the resulting assignments are distilled into a context-agnostic bigram segmentation model for efficient inference (Libovický et al., 2024).

In retrieval (KinyaColBERT) and encoder architectures, morphologically grounded tokens are fed into lower-tier transformer layers, with morphemes, POS tags, and morphosyntactic features all embedded and pooled; these produce interpretable and fine-grained word-level encodings for document and query representations (Nzeyimana et al., 4 Jul 2025).

5. Empirical Evaluation and Task-Specific Benefits

Improvements are observed on both intrinsic and extrinsic metrics:

• Segmentation Precision/Recall/F₁: Lexically grounded methods reliably yield higher morpheme-boundary precision and F₁—for example, MorphMine outperforms Morfessor and BPE by 6–30 F₁ points across English, German, and Turkish; boundary precision increases by ≥5–10 pp with explicit morphological pre-tokenization (Libovický et al., 2024, El-Kishky et al., 2019).
• Downstream Tasks: Across machine translation, POS tagging, language modeling, and low-resource retrieval, consistent improvements are reported. For example, in POS tagging, combined Morfessor+BPE+embedding achieves up to 1.2 pp gain in Hungarian and best average accuracy in a multilingual setting (Libovický et al., 2024); KinyaColBERT achieves +26.2 pp MRR@10 over WordPiece on Kinyarwanda retrieval (Nzeyimana et al., 4 Jul 2025).
• Fertility, Perplexity, and BLEU: MorphTok+CBPE reduces token fertility by up to 1.7%, improves LLM perplexity (up to −14.3%), and increases BLEU/ChrF/COMET in Hindi–Marathi MT (Brahma et al., 14 Apr 2025).
• Human Evaluation: MorphTok’s human-grounded EvalTok reveals a >23% increase in perfect morpheme-aligned segmentations over standard BPE (Brahma et al., 14 Apr 2025).

6. Limitations and Open Challenges

Key weaknesses of dictionary-driven approaches remain: incompleteness and coverage gaps (e.g., DeriNet's partial coverage yields few or no splits for many words); rigidity in rapidly evolving, colloquial, or code-mixed domains; and heavy up-front human curation (Macháček et al., 2018, 2207.13333). Hybrid and neural approaches mitigate these by relaxing hard boundaries and allowing for distributional generalization or fallback, yet risk introducing spurious units if forced to back off too aggressively to character-level or BPE segmentations.

For highly morphophonological languages, sandhi and phonological alternations complicate naive substring-based morph detection, motivating script-specific or phonology-aware constraints (e.g., CBPE in Devanagari, sandhi splitting in Sanskrit and Hindi) (Brahma et al., 14 Apr 2025). Performance on extremely agglutinative or polysynthetic languages remains a persistent research frontier.

A plausible implication is that future robust tokenization in LLM and retrieval pipelines for low-resource and morphologically complex languages will require the systematic integration of both symbolic (dictionary/grammar) and learned (neural, distributional) segmentation, with dynamic updating and multilingual alignment.

7. Extensions and Prospects

Several avenues for advancement are identified in the literature:

• Hybrid models: Joint optimization that combines rule-driven boundary constraints with unsupervised or lexically informed statistical segmentation (e.g., BPE scoring with penalty for crossing dictionary-derived boundaries or bigram probabilities initialized from embedding-driven splits) (Libovický et al., 2024, Wu et al., 2018).
• Script- and Phonology-Aware Segmentation: CBPE and analogous methods for Hebrew, Arabic, Sanskrit, and other languages with non-trivial morphophonemics and diacritics (Brahma et al., 14 Apr 2025).
• Unsupervised Discovery: Enhanced neural segmental models that can dynamically grow or prune the lexicon, learn morphophonological correspondences, or integrate multilingual priors to promote subword alignment and resource sharing (Kawakami et al., 2018, El-Kishky et al., 2019).
• Token Interpretability and Parameter Efficiency: Grounded tokenization directly reduces model parameterization (smaller, reusable embedding tables) and improves interpretability for downstream explainability and error analysis (Brahma et al., 14 Apr 2025, Nzeyimana et al., 4 Jul 2025).

The field demonstrates a convergence towards integrated, linguistically aware NLP systems in which all layers—tokenization, embedding, model architecture—cooperate to align atomic units with underlying lexical structure, promoting efficiency and generalization especially for languages and domains underrepresented in large-scale corpora.