Achieving Tokenizer Flexibility in Language Models through Heuristic Adaptation and Supertoken Learning

Abstract: Pretrained LLMs are often constrained by their fixed tokenization schemes, leading to inefficiencies and performance limitations, particularly for multilingual or specialized applications. This tokenizer lock-in presents significant challenges. standard methods to overcome this often require prohibitive computational resources. Although tokenizer replacement with heuristic initialization aims to reduce this burden, existing methods often require exhaustive residual fine-tuning and still may not fully preserve semantic nuances or adequately address the underlying compression inefficiencies. Our framework introduces two innovations: first, Tokenadapt, a model-agnostic tokenizer transplantation method, and second, novel pre-tokenization learning for multi-word Supertokens to enhance compression and reduce fragmentation. Tokenadapt initializes new unique token embeddings via a hybrid heuristic that combines two methods: a local estimate based on subword decomposition using the old tokenizer, and a global estimate utilizing the top-k semantically similar tokens from the original vocabulary. This methodology aims to preserve semantics while significantly minimizing retraining requirements. Empirical investigations validate both contributions: the transplantation heuristic successfully initializes unique tokens, markedly outperforming conventional baselines and sophisticated methods including Transtokenizer and ReTok, while our Supertokens achieve notable compression gains. Our zero-shot perplexity results demonstrate that the TokenAdapt hybrid initialization consistently yields lower perplexity ratios compared to both ReTok and TransTokenizer baselines across different base models and newly trained target tokenizers. TokenAdapt typically reduced the overall perplexity ratio significantly compared to ReTok, yielding at least a 2-fold improvement in these aggregate scores.

• The paper introduces TokenAdapt, a model-agnostic framework that employs heuristic adaptation and supertoken learning to overcome tokenizer lock-in.
• It combines local heuristic sub-token decomposition with a k-nearest-neighbor global approach to initialize embeddings while preserving semantic relationships.
• Experimental results demonstrate a reduction in perplexity ratios up to 2-fold, highlighting improved tokenization efficiency over baseline methods.

Achieving Tokenizer Flexibility in LLMs through Heuristic Adaptation and Supertoken Learning

Introduction

The paper "Achieving Tokenizer Flexibility in LLMs through Heuristic Adaptation and Supertoken Learning" addresses the issue of tokenizer lock-in in LLMs, which restricts their flexibility and efficiency, especially in multilingual or specialized domains. The standard subword tokenization schemes often lead to inefficiencies such as token fragmentation, impacting semantic fidelity and computational costs. Efforts to extend vocabularies through continued pre-training are resource-intensive and do not necessarily resolve inefficiencies inherent in original tokenization strategies. The authors propose TokenAdapt, a model-agnostic framework that uses heuristic adaptation and novel supertoken learning to achieve tokenizer flexibility.

TokenAdapt Framework

Heuristic-Based Initialization

TokenAdapt introduces a hybrid heuristic approach to initialize embeddings for unique tokens within a new tokenizer. This heuristic combines local and global strategies to preserve semantic relationships.

• Local Heuristic: This involves decomposing new tokens into sub-tokens using the old tokenizer. Semantic similarities between these sub-tokens and the new token are calculated. These similarities are used to weight the combination of original embeddings, normalized by sub-token length.

  Figure 1: Core logic of the Local Heuristic.

• Global Heuristic: Utilizes a k-nearest-neighbor (kNN) approach in an auxiliary embedding space to find semantically similar tokens in the original vocabulary. These neighbors' embeddings are weighted based on similarity scores.

  Figure 2: Core logic of the Global Heuristic.

• Hybrid Integration: The final embedding of a new token is derived from a weighted combination of the local and global heuristics. This approach aims to project the token into the model's embedding space while maintaining crucial semantic relationships.

  Figure 3: Core logic of the Local and Global Heuristics respectively. This diagram illustrates the two main pathways (Local and Global) for generating components of a new token's embedding, which are then combined via Hybrid Integration.

Supertoken Learning

In addition to heuristic transplantation, the framework introduces supertoken learning to improve tokenization efficiency. Supertokens are multi-word tokens that enhance compression and reduce fragmentation by enabling token mergers that align more closely with semantic units.

Experimental Evaluation

The authors empirically validate TokenAdapt by demonstrating its superior zero-shot perplexity performance compared to baselines like ReTok and TransTokenizer. Across target domains, TokenAdapt consistently provided lower perplexity ratios, reflecting better semantic preservation immediately post-transplantation. The hybrid heuristic approach showed significant improvements by reducing perplexity ratios up to 2-fold compared to other methods.

Conclusion

TokenAdapt represents an efficient and effective solution for enabling tokenizer flexibility in LLMs without extensive retraining. By utilizing a hybrid heuristic for token transplantation and integrating the learning of supertokens, TokenAdapt addresses both vocabulary extension and tokenization efficiency, adapting pre-trained models to new domains and languages with minimal overhead.

Future work could explore more adaptive weighting strategies within the heuristic framework, alternative auxiliary embedding spaces, and improved coordination between supertoken and transplantation methodologies. This research provides a scalable approach to the emerging challenges facing adaptable natural LLMs.