In-Place Tokenizer Expansion for Pre-trained LLMs

Abstract: A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time. When those priorities shift, languages added later are split into many more tokens per word, which can raise latency, compute, and energy consumption for users of those languages. Cloud models can afford a broad vocabulary because the embedding and LM-head matrices are a small fraction of their parameters. On a compact model those matrices are a material share of per-token decode bandwidth, so on-device models ship small vocabularies and accept fragmentation outside a fixed language set. We present tokenizer expansion, an in-place recipe for upgrading a pre-trained model's tokenizer when the model producer controls its design. We continue the existing tokenizer's BPE merges on a multilingual corpus, so most source tokens carry over unchanged as single tokens and every new token has an exact decomposition into source tokens. We copy the carried-over embedding rows unchanged and initialize new rows as the mean of their source sub-token embeddings. A two-stage adaptation, embedding-only training then full-model continued pre-training, recovers source-checkpoint quality. We apply the recipe to a continued pre-trained checkpoint of LFM2-8B-A1B, an 8B-parameter Mixture-of-Experts model, to help produce LFM2.5-8B-A1B with a 128K tokenizer. The expanded tokenizer encodes Hindi and Vietnamese in roughly $2.4\times$ and $2.6\times$ fewer tokens than the source (up to $4.0\times$ on Thai). Combining these reductions with the measured per-token cost of the larger vocabulary, we estimate a $2.2$-$3.7\times$ per-character decode speedup for these languages across our reference devices. We release the model weights and the expanded tokenizer, and report the negative findings that shaped the recipe.

• The paper introduces an in-place tokenizer expansion method that extends pre-trained LLM vocabularies via continued BPE merges, reducing token fragmentation in underrepresented languages.
• It employs a two-stage adaptation pipeline combining embedding-only training and full-model continued pre-training to preserve embedding norms and recover model quality.
• Empirical results show significant token count reductions, decode speedups, and performance improvements across diverse multilingual benchmarks.

In-Place Tokenizer Expansion for Pre-trained LLMs: Extension, Adaptation, and Efficiency

Motivation and Context

Tokenizers fixed at pre-training time create vocabularies reflecting corpus priorities. As deployment shifts focus, underrepresented languages in the original corpus suffer token fragmentation—each word splits into many tokens—leading to increased latency, compute, and energy for those languages. In large cloud models, embedding and LM-head matrices are negligible fractions of parameter count, enabling large vocabularies that cover broad scripts. For compact, on-device models, however, embedding and output projections are significant relative to overall size and can dominate per-token decode bandwidth, necessitating smaller vocabularies with priority language coverage and acceptance of fragmentation elsewhere.

This paper focuses on efficient in-place tokenizer expansion for pre-trained LLMs when the model producer controls the tokenizer design. The method allows broadening token coverage—especially for languages that were previously under-tokenized—without retraining models from scratch or relying on complex zero-shot transfer solutions.

Expansion Algorithm and Embedding Initialization

The central contribution is an in-place expansion recipe based on continuing the BPE merge process for the existing tokenizer on a carefully weighted multilingual corpus, adding merges for underrepresented languages while keeping prior merges fixed. Most source tokens carry over one-to-one; new tokens decompose exactly into source tokens. This property allows direct embedding row transfer for carried-over tokens and facilitates initialization of new rows as the mean of their constituent source embeddings.

Formally, for each new token $t$ in the expanded vocabulary ($V_{\text{new}}$), its embedding is initialized as:

$$E_{\text{new}}[t] = \frac{1}{N} \sum_{i=1}^N E_{\text{src}}[t'_i]$$

where each $t'_i$ is a source token in the deterministic decomposition. This preserves embedding norms and avoids random initialization, aligning with established empirical practices for vocabulary expansion [hewitt2021init, mundra2024empirical].

Adaptation Pipeline

The adaptation proceeds in two dedicated stages:

Stage 1 (Embedding-only Training):

Train only newly introduced embedding rows (corresponding to expanded tokens) with the model body frozen. The source embedding rows transferred directly are also kept frozen, avoiding detrimental drift. The data mixture is weighted toward under-tokenized languages. This stage recovers most quality lost from the zero-shot swap.

Stage 2 (Full-model Continued Pre-training):

Unfreeze all parameters and continue training on a balanced multilingual corpus, now tokenized with the expanded tokenizer. This stage integrates new tokens into the model and rebalances capabilities across languages, recovering remaining quality and infusing improved coverage.

After these, the standard mid-training (with extended context length) and post-training take place, fully restoring and advancing aggregate performance.

Numerical Results and Efficiency Gains

Tokenizer expansion applied to LFM2-8B-A1B produces LFM2.5-8B-A1B with a 128K-token vocabulary. The following strong numerical results are highlighted:

• Token-count reductions:
  • Hindi: $2.4\times$ fewer tokens per word
  • Vietnamese: $2.6\times$ fewer tokens per word
  • Thai: up to $4.0\times$ fewer tokens
• Decode speedup:

Combining measured per-token costs (due to a larger vocabulary matrix) and per-language compression, the estimated per-character decode speedup for these languages ranges from $2.2$–$3.7\times$ across tested devices.

• Aggregate quality recovery:

After expansion and adaptation, the model matches or exceeds source checkpoint performance across eight public benchmarks: knowledge, math, code, and multilingual tasks. Stage~2 exceeds source by $+3.6$ aggregate points, with per-language improvements on Global-MMLU (e.g., Vietnamese $V_{\text{new}}$0, Hindi $V_{\text{new}}$1).

• Preservation of prior language quality:

Languages efficiently tokenized in the source retain near parity or minor improvements, despite the larger vocabulary incurring a uniform per-token cost (up to $V_{\text{new}}$2 decode regression per character for English).

Ablations and Empirical Findings

Embeddings freezing:

Training all rows (rather than just new ones) during Stage~1 degrades performance, particularly for generative tasks, and induces output repetition.

Data mixture:

Unbalanced mixtures in Stage~2 (predominantly English) cause broad performance collapse in generative tasks, even for English, while multiple-choice accuracy remains unchanged or improved.

Both findings emphasize the need for rigorous evaluation using generative benchmarks, not just multiple-choice log-probability.

Implications and Future Directions

This recipe provides a practical approach to cross-lingual vocabulary adaptation for production-grade on-device models, offering an alternative to retraining or complex transfer schemes. The continued-BPE method leverages deterministic token decompositions to facilitate straightforward embedding initialization and efficient adaptation.

Future directions may include miniaturizing the continued pre-training budget, further optimizing compression-vs-vocabulary trade-offs, and extending the recipe to other architectures or off-the-shelf tokenizers as decomposition/transfer methods mature [minixhofer2024zett, goddard2025omp]. There is also scope for automatic balancing of vocabulary allocations in response to evolving deployment requirements and for evaluating compression gains as vocabulary size increases past 128K—where device constraints become dominant.

Conclusion

In-place tokenizer expansion, as formulated, allows efficient coverage extension for pre-trained LLMs while preserving prior capabilities. The recipe is validated at MoE scale and delivers substantial gains in token efficiency and decoding speed for under-tokenized languages, with minimal regressions for others. The public release of model weights and expanded tokenizers solidifies its utility for practical LLM deployment.

Limitations

• Applicability restricted to cases where the original tokenizer's merges and configuration are available, and continued-BPE training is feasible.
• The recipe incurs a non-trivial continued pre-training compute cost.
• Uniform per-token cost due to vocabulary expansion can incur slight regressions for previously well-tokenized languages.

Practical Availability

Weights and tokenizers for LFM2.5-8B-A1B are openly available, supporting deployment across mainstream inference frameworks.

---

References:

• "In-Place Tokenizer Expansion for Pre-trained LLMs" (2607.15232)
• Related works: [mundra2024empirical], [yamaguchi2024cva], [minixhofer2024zett], [goddard2025omp], [hewitt2021init], [purason2025teaching]