Downstream performance impact of SuperBPE at Trinity’s experimental scale

Determine whether the SuperBPE tokenizer—trained by first using standard whitespace splitting, truncating, and then resuming training without the whitespace constraint to learn multi‑word tokens—yields measurable downstream performance improvements for Trinity language models at the experimental scale considered, beyond the demonstrated token compression gains.

Background

In evaluating tokenizer choices, the authors trained a 200,000‑token BPE vocabulary and compared it against a SuperBPE variant that learns multi‑word tokens by resuming training without the whitespace constraint. Although SuperBPE delivered substantially better compression (fewer tokens per document), the authors did not observe a corresponding improvement in downstream model performance at their experimental scale.

This leaves unresolved whether SuperBPE’s compression benefits translate into actual capability gains for LLMs within the Trinity training pipeline and resource regime. Clarifying this would inform tokenizer design trade‑offs for future Trinity iterations and similar large‑scale pretraining efforts.

References

While our SuperBPE variant achieved substantially better compression — particularly on English text (~29% fewer tokens) and reasoning traces (~27% fewer tokens) — we were unable to reproduce a corresponding improvement in downstream model performance at our experimental scale.

— Arcee Trinity Large Technical Report  (2602.17004 - Singh et al., 19 Feb 2026) in Subsubsection “Vocabulary Size” under “Tokenizer” (Architecture, Section 2)