Next-Character Sampling Bias in Neural Language Models

• Next-character sampling bias is the systematic deviation from the ideal character-level distribution due to architectural, tokenization, and training/inference mismatches.
• Empirical metrics such as PER, WER, perplexity, and cross-entropy highlight its impact on error propagation and the fidelity of generated content.
• Mitigation approaches like loss-based sampling, Branch & Pass, and ByteSampler improve model accuracy while balancing computation and decoding strategies.

Next-character sampling bias refers to systematic deviations—often irreducible—from an ideal character-level distribution when sampling in autoregressive neural LLMs. This phenomenon arises due to architectural choices (such as subword tokenization and stateful decoding) and training/inference mismatches in character-level or byte-level decoders. Next-character sampling bias is intrinsically linked to exposure bias and tokenization bias, and has significant effects on the fidelity, diversity, and accuracy of generated text, phoneme sequences, or code.

1. Characterization and Sources of Next-Character Sampling Bias

Next-character sampling bias emerges whenever the probability assigned to the next character $c$ given a context $x_1^n$ deviates from the probability distribution that would be realized by a truly character-level model. In autoregressive frameworks, such as transformers or RNNs, this bias is compounded by two mechanisms:

• Exposure bias: During training with teacher forcing, models always condition on gold-standard prefixes $y_0{:}i-1$. At inference, the context consists of self-sampled predictions $\hat{y}_0{:}i-1$, magnifying downstream errors, particularly on longer sequences (Yoon et al., 2023).
• Tokenization bias: Models operating at the subword unit level (BPE or MPE) do not align their token boundaries with the underlying character-level transitions, so the naive next-character distribution $$p_\text{tok}(c|x_1^n) = \sum_{t:\,\mathrm{decode}(t)_1 = c} p(t|t_1^i)$$ systematically differs from the true distribution $p_\text{true}(c|x_1^n)$, resulting in $\Delta(c|x_1^n) \neq 0$ even with perfect training (Phan et al., 2024).

In character-level decoders (e.g., ByT5), each grapheme may expand into multiple bytes, substantially increasing sequence length and the opportunities for error propagation. Small mistakes early in generation shift the context for future predictions, causing steep drops in accuracy and compounding bias as output grows (Yoon et al., 2023).

2. Quantitative Effects and Empirical Metrics

Next-character sampling bias manifests in both direct metrics and secondary effects:

• Phoneme Error Rate (PER)/Word Error Rate (WER): In sentence-level G2P, loss-based sampling improved PER by ~2.06–2.15 percentage points and WER by ~1.37–1.80 points over teacher forcing for short and long test sets (Yoon et al., 2023).
• Perplexity and Cross-Entropy: At the character level, sampling schemes substantially impact test-set perplexity and cross-entropy, revealing the degree of bias imposed by stateful generation (Boom et al., 2018, Hayase et al., 17 Jun 2025).
• Bias formalism: $$\Delta(c|x_1^n) = p_\text{tok}(c|x_1^n) - p_\text{true}(c|x_1^n)$$, with empirical Markov chain experiments demonstrating mean-squared error of 0.185 (biased) versus 0.001 (unbiased) in transition recovery (Phan et al., 2024).

Character-level RNN experiments show that windowed sampling schemes yield lower perplexity and higher fidelity to real token histograms, while progressive sampling introduces drift, under-representing lower-probability symbols due to accumulated errors (Boom et al., 2018). Byte-level methods such as ByteSampler demonstrate near-optimal next-character accuracy (English: 81.6%, Chinese: 52.7%) with minimal computational overhead versus naive token-based approaches (Hayase et al., 17 Jun 2025).

3. Mechanisms and Variants of Sampling Bias

Sampling bias is tightly linked to the choice of decoding algorithm and state management:

• Greedy Sampling ($T=1$): Tends to collapse entropy, favoring the most probable token and under-representing rare characters (Boom et al., 2018).
• Ancestral Sampling: Reflects expected diversity based on the model’s learned distribution, but still inherits mismatches from tokenization (Boom et al., 2018, Phan et al., 2024).
• Top-k/Nucleus (top-p) Sampling: Imposes artificial cutoffs, excluding long-tail tokens and further distorting statistics, with controllable impacts via temperature scaling (Boom et al., 2018).
• Windowed vs. Progressive State Management: Windowed resets hidden states, preserving true frequency distributions; progressive updates inherit and propagate earlier mistakes, amplifying bias especially in longer generations (Boom et al., 2018).

In byte-level decoders, variable-length tokens and UTF-8 expansion multiply bias amplification steps. Prompt boundary issues further distort predicted byte distributions when the input does not align with token boundaries, as observed in code and multilingual scenarios (Hayase et al., 17 Jun 2025).

4. Mitigation Strategies for Next-Character Sampling Bias

Several advanced algorithms address bias without requiring retraining:

• Loss-based sampling (Exposure Bias mitigation): Selectively corrupts the decoder’s gold input at positions with highest cross-entropy loss, training the model to recover from its own errors. Adaptively tunes the corruption ratio based on validation PER, yielding flatter error accumulation and improved robustness (Yoon et al., 2023).
• Branch & Pass (Tokenization Bias mitigation): Recursively refactors the next-character probability as a sum over valid partial-token events, enabling unbiased reconstruction of $p_\text{true}(c|x_1^n)$ from a tokenized LM via a linear number of extra model calls (Phan et al., 2024).
• ByteSampler (Prompt Boundary Problem): Wraps a BPE-tokenized model to sample exactly from the character or byte-level distribution using a "Valid Covering Tree" (VCT), solving prompt-boundary issues and enabling ensemble/proxy-tuning across model vocabularies (Hayase et al., 17 Jun 2025).

Uniform random sampling during training offers some reductions in exposure bias, but consistently underperforms compared to loss-based or adaptive methods. In all cases, aligning training and inference contexts is essential to prevent runaway error accumulation (Yoon et al., 2023).

5. Practical Implications, Limitations, and Open Challenges

Sampling bias directly impacts real-world applications—sentence/paragraph-level G2P, multilingual text generation, program synthesis, and model ensembling:

• Usability: Character-level or byte-level G2P improves heteronym/linking sound handling, but only if sampling bias is controlled (Yoon et al., 2023).
• Interoperability: ByteSampler's unification of vocabularies enables ensembling across models with distinct tokenizers, facilitating proxy-tuning and cross-model transfer (Hayase et al., 17 Jun 2025).
• Limitations: Branch & Pass and ByteSampler correctness proofs currently cover deterministic BPE/MPE only; extension to unigram or regex-based pretokenization remains open (Phan et al., 2024, Hayase et al., 17 Jun 2025).
• Computational Overhead: Two-pass or streaming tokenization increases cost (e.g., ≈0.7 extra token calls/byte with ByteSampler), and bytewise sampling is 4–5× slower than tokenwise (Hayase et al., 17 Jun 2025).
• Decoding Algorithms: Exact bias mitigation applies fundamentally to ancestral sampling; adaptation for greedy/top-k/top-p decoding is an unresolved research area (Hayase et al., 17 Jun 2025).

Experiments confirm that Markov-like dynamics and true character distributions are only faithfully recovered by unbiased estimators; naive token-based sampling remains irreducibly biased regardless of model scale or training data (Phan et al., 2024).

6. Guidance and Trade-Offs in Sampling Scheme Selection

Paper-guided recommendations include:

• Multi-loss training is more efficient and converges faster than single-loss (Boom et al., 2018).
• Avoid cross-context hidden state transfers to prevent degeneration and instability (Boom et al., 2018).
• For maximum fidelity to training data, use windowed sampling with multi-loss aggregation (Boom et al., 2018). For exploration or diversity, controlled temperature or nucleus sampling may optionally be layered, with explicit trade-offs in entropy and perplexity.
• In G2P and extended character-level prediction tasks, loss-based or adaptive sampling is preferred over uniform corruptions for exposure bias mitigation (Yoon et al., 2023).

A plausible implication is that future advances must generalize unbiased next-character sampling to all tokenization schemes and aggressive context management strategies while preserving efficiency.

7. Historical Evolution and Emerging Directions

Recent research highlights the persistent and universal nature of next-character sampling bias across model classes and application domains. Initial work focused on RNNs and sequential softmax decoding (Boom et al., 2018), but the growth of subword tokenization in large models (BPE, MPE) and byte-level transformers (e.g., ByT5) has brought new urgency and opportunities for algorithmic solutions (Yoon et al., 2023, Phan et al., 2024, Hayase et al., 17 Jun 2025).

Ongoing challenges include theoretical extension beyond BPE, computational speedups for bytewise generation, integration with speculative decoding, and mitigation of decoding bias in non-sampling (greedy/top-p) regimes. Empirical evidence consistently confirms that direct, unbiased estimators restore fidelity and diversity unattainable by naive token-based sampling, solidifying next-character bias as a central topic in large-scale language modeling.