BoundlessBPE: Enhancing Subword Tokenization

• BoundlessBPE is a modification of BPE that relaxes pretoken segmentation to merge adjacent tokens into 'superwords', enhancing vocabulary flexibility.
• It employs a scoring mechanism for both standard merges and supermerges while dynamically deleting tokens based on an IoS threshold, improving compression.
• Empirical results show at least a 21% improvement in token distribution uniformity and nearly 20% better bytes per token compared to standard methods.

BoundlessBPE is a modification of the standard Byte Pair Encoding (BPE) algorithm that addresses the limitations imposed by the pre-tokenization step in contemporary text tokenization pipelines. Standard BPE, widely adopted for subword tokenization in LLMs, operates on text already segmented into discrete units known as pretokens (typically delimited by whitespace or punctuation). This pre-segmentation, while practical for encouraging whole-word token coverage, introduces an artificial frequency bias that concentrates vocabulary capacity on common full-length words and constrains subsequent subword merges primarily to rare or idiosyncratic forms. BoundlessBPE proposes to relax pretoken boundaries by enabling merges across pretoken boundaries—thereby constructing "superwords"—and incorporating dynamic vocabulary management through selective deletions, resulting in substantially more uniform token frequency distributions and increased compression efficiency (Schmidt et al., 31 Mar 2025).

1. Standard BPE with Pre-tokenization

In standard pipelines, the raw corpus is initially split into pretokens via a Unicode‐aware regular expression. Each pretoken, representing word-like spans (e.g., " the", "car", ","), is then further decomposed into its constituent bytes. The merge process proceeds on these bytes within each pretoken independently:

• Initialization: Every pretoken is represented as a sequence of bytes.
• Iterative Merging: At each iteration, adjacent token pairs across the corpus are counted, the most frequent pair is merged, and this step is repeated until the vocabulary size $|V|$ reaches a preset limit.

This process rapidly encodes frequent full-word pretokens as single tokens. Subsequent merges occur almost exclusively within rare or morphemically complex words, restricting vocabulary growth and producing a sharp, Zipfian head in the token-frequency distribution.

2. BoundlessBPE: Algorithmic Innovations

BoundlessBPE introduces two interdependent extensions:

A. Supermerges

• Definition: A "supermerge" considers merging two adjacent pretokens (e.g., " of" and " the") into a superword (e.g., " of the") provided both are already represented as single tokens.
• Scoring: Supermerges are scored identically to standard BPE merges via adjacency counts $c(p_i, p_{i+1})$ and directly compete with ordinary merges each iteration.
• Constraints: Only pairs matching a word-like regular expression (BOUNDLESS_PATTERN) are permitted as candidates, precluding merges that would create non-linguistic or semantically incoherent tokens (such as punctuation+word).

B. PickyBPE-style Deletions

• After a regular merge, the Intersection-over-Self (IoS) criterion is calculated for each source token:
  • $$\mathrm{IoS}(t_1 \rightarrow t_1t_2) = c(t_1, t_2) / f(t_1)$$,
  • $$\mathrm{IoS}(t_2 \rightarrow t_1t_2) = c(t_1, t_2) / f(t_2)$$.
• If the IoS exceeds a threshold $\tau = 0.9$ for either token, that token is deleted from the vocabulary and resplit into bytes, recycling capacity for more frequently co-occurring tokens.

At each iteration, the candidate operations (regular merges, supermerges, and IoS-triggered deletions) are scored, the single best-scoring operation is applied globally, and the process repeats until the vocabulary budget is exhausted.

3. Token Distribution and Uniformity

To quantify the uniformity of the resulting token-frequency distribution, BoundlessBPE adopts Rényi-efficiency as introduced by Zouhar et al. This metric is defined as

$$H_\alpha(P) = \frac{1}{1-\alpha} \log \sum_i p_i^\alpha\,, \qquad \mathrm{Eff}_\alpha(P) = \frac{H_\alpha(P)}{\log |V|},$$

where higher $\mathrm{Eff}_\alpha$—with $\alpha=2.5$ as recommended—indicates greater uniformity. Across four vocabulary sizes (40,960 to 131,072), BoundlessBPE achieves at least a 21% higher $\mathrm{Eff}_{2.5}$ than standard BPE, WordPiece, or UnigramLM baselines.

Empirical analysis shows BoundlessBPE creates a higher-frequency tail (many more mid-frequency tokens) and a less dominant head (lower maximum frequencies among the most common tokens). Over 97.5% of the constructed vocabulary appears at least once in held-out text, compared to 82–95% for other strategies, indicative of a more practically useful and less redundant lexicon.

4. Compression and Inference Efficiency

Compression efficiency is assessed via bytes per token (BPT):

$$\mathrm{BPT} = \frac{\text{total bytes in corpus}}{\text{number of tokens produced}}.$$

BoundlessBPE improves BPT by at least 19.7% relative to the strongest baseline for all evaluated vocabulary sizes (e.g., BPT $\approx$ 4.6 for BoundlessBPE vs. $\approx$ 3.8 for baseline BPE at $|V|=131,072$). This translates directly to reduced sequence lengths and thus lower computational and memory overhead in transformer-based architectures, which are sensitive to token count due to self-attention complexity scaling as $O(T^2)$ with sequence length $T$. While downstream performance improvements await empirical validation on model quality metrics such as perplexity or fine-tuning loss, the compression gains are immediate and measurable.

5. Limitations and Prospective Extensions

Several caveats and future research directions are identified:

• Training Overhead: Supermerges require consideration of adjacency across pretokens, precluding straightforward aggregation and resulting in significant computational cost (approximate 4.7 CPU-days to train on 1 GB of data). Efficient data structures or streaming algorithms may mitigate this bottleneck.
• Controlled Deletions: PickyBPE deletions are currently disabled for superwords to avoid implementation complexity. Enabling this feature could further flatten the long-tail of the token distribution.
• Hyperparameter Selection: The IoS threshold ($\tau=0.9$) and supermerge regex are determined heuristically. Systematic search or adaptive tuning may yield marginal improvements.
• Downstream Performance: No direct evaluation of LLM perplexity or fine-tuning outcomes has been conducted. The anticipated inferential speedup and vocabulary utility must be corroborated in model training contexts.
• Language Coverage: The method and current BOUNDLESS_PATTERN are English-centric. Adaptation for morphologically rich or non-Latin-script languages remains an open research avenue.

6. Significance in Tokenization Research

BoundlessBPE demonstrates that relaxing the architectural pretokenization boundary inherent in modern subword modeling pipelines can yield quantifiably flatter token distributions and appreciably higher compression rates. Its methodology remains compatible with bottom-up learning mechanisms characteristic of BPE, offering a theoretically simple yet practically impactful extension for LLM pretraining and inference pipelines. The algorithm is readily incorporable into existing frameworks, delivering immediate efficiency yields and establishing a new direction for tokenization research beyond the customary word-boundary constraint (Schmidt et al., 31 Mar 2025).