Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index

Abstract: Reinforcement learning (RL) has become a powerful tool for propelling LLMs beyond imitation-based training towards more robust reasoning capabilities. Among existing approaches, RL with Verifiable Rewards (RLVR) has emerged as a pivotal paradigm for advancing LLM reasoning. Despite its empirical success, recent studies have offered different insights. One line of inquiry advocates prioritizing high-entropy token positions during training, while another perspective cautions against allowing low-probability tokens to dominate gradient updates. Notably, although high-entropy tokens are usually correlated with low probability, both paradigms empirically yield substantial performance gains. In this work, we argue that evaluating sampled-token probability or entropy in isolation is insufficient to capture the policy optimization dynamics. To resolve this tension, we introduce the Relative Surprisal Index (RSI), a principled, information-theoretic metric that naturally couples the token's entropy with the probability of the selected token. We show that, under mild conditions, RSI is related to the local ratio between the first-order variations of the logit-gradient norm and predictive entropy under a selected-logit perturbation. Building on RSI, we propose RSI Selection (RSI-S), an entropy-adaptive token filtering method that retains tokens within a stable RSI interval. RSI-S successfully reconciles previous contradictory paradigms and filters out both redundant low-surprisal tokens and unstable high-surprisal tail tokens. Empirical evaluations show that RSI-S achieves higher avg@32 accuracy across different model scales (Qwen2.5-1.5B, 3B, and 7B) on AIME and AMC benchmarks: RSI-S improves avg@32 accuracy by 2--3 percentage points over GRPO. Overall, RSI offers a promising perspective for RLVR improvement.

• The paper introduces the Relative Surprisal Index (RSI) for token selection in RLVR, unifying entropy and probability into a principled metric.
• It proposes RSI-S, an interval-based filtering method that dynamically adapts token selection to achieve consistent performance improvements.
• Empirical studies show RSI-S improves accuracy by 2–3 points and reduces threshold sensitivity across multiple model scales for math benchmarks.

Adaptive Token Selection in RLVR via the Relative Surprisal Index

Introduction

Token-level granularity in reinforcement learning with verifiable rewards (RLVR) for LLMs has driven recent debates on which tokens should be prioritized during optimization. Prevailing approaches have taken either an entropy-centric view—advocating focus on high-entropy tokens [wang2026beyond]—or argued for suppression of low-probability tokens to avoid unstable gradient updates [yang2025not]. The paper "Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index" (2606.31575) introduces the Relative Surprisal Index (RSI), a metric that couples entropy and sampled-token probability, providing a unified, theoretically grounded criterion for token selection in RLVR. The RSI framework, with its practical instantiation RSI-S, demonstrates consistent and significant performance gains across multiple model scales and mathematical reasoning benchmarks.

Figure 1: The RSI framework: computation of token-wise entropy and probability, interval-based token selection via RSI-S, and integration into RL pipelines yielding improved stability and accuracy.

Theoretical Foundations of the Relative Surprisal Index

The central contribution is the formulation of RSI, defined for each token $o_t$ (sampled under $\pi_\theta(\cdot \mid q, o_{<t})$) as:

$$\mathrm{RSI}_t = 1 + \frac{ \log \pi_\theta(o_t \mid q, o_{<t})}{ \mathcal{H}_t }$$

where $\mathcal{H}_t$ is the entropy of the predictive distribution at timestep $t$. RSI thus measures the deviation of the realized token’s log-probability from the average log-probability, normalized by uncertainty.

Crucially, the paper establishes that RSI is mathematically related to the ratio of first-order variations between the logit-gradient norm and entropy under a logit perturbation (Theorem 1). This connection grounds RSI as an analytically principled token-importance signal—not merely a heuristic blending of entropy and probability.

The RSI distribution is shown to be highly concentrated near $1$, indicating that the majority of generated tokens are in low-surprisal regimes, with only a minority being likely candidates for significant optimization (Figure 2, right). Dependence structure and scale-invariance across model sizes are also validated.

Figure 2: Left: Dominance of high-probability tokens at low entropy; Middle: Stable negative Spearman correlation between token entropy and probability; Right: Empirical density of RSI, showing heavy concentration near $1$ (low surprisal).

RSI Selection (RSI-S): Entropy-Adaptive Token Filtering

The paper operationalizes RSI via RSI Selection (RSI-S), an interval-based filtering process that retains only those tokens whose RSI falls within $[a, b]$:

• The upper bound $b<1$ excludes highly predictable, low-surprisal tokens deemed redundant for optimization.
• The lower bound $a<0$ prunes unstable, extremely low-probability (high-surprisal) tail tokens that would destabilize updates.

This entropy-adaptive interval replaces both rigid probability-based cuts and fixed entropy thresholds, yielding a dynamic method responsive to the current policy’s local uncertainty.

A key analytical result is that for each token, the valid probability eligibility is scaled by the entropy, with the adaptation:

$\pi_\theta(\cdot \mid q, o_{<t})$0

This entropy-guided scaling enables selective attention to moderately surprising tokens, balancing exploration and stability.

Empirical Results and Analysis

Extensive experiments on base Qwen2.5 models (1.5B, 3B, and 7B) trained on DAPO-MATH-17K and evaluated on AIME and AMC contest subsets substantiate the effectiveness of RSI-S over entropy-based and probability-based baselines. Notably, RSI-S yields avg@32 accuracy improvements of 2–3 points and shorter generations compared to the GRPO baseline.

Figure 3: Left: Training accuracy curves with GRPO vs. GRPO+RSI-S over steps; Right: Evaluation accuracy across six math benchmarks and model sizes—RSI-S outperforms all baselines consistently.

Ablation studies indicate:

• Superiority over Static Thresholding: Static probability filtering is significantly less effective and less robust than RSI-S, which adapts dynamically to local distributional uncertainty.
• Necessity of Both Bounds: Removing either the upper or lower RSI bound degrades accuracy and stability, confirming the theoretical motivation for a two-sided filter.
• Model-Scale Adaptation: Larger LLMs benefit from a lower RSI upper bound, consistent with enhanced exploration capability, while smaller models require more conservative filtering.

The recall of so-called RL-critical tokens—defined as those with high pre/post-RL Jensen--Shannon divergence—is also higher and more robust for RSI-based selection relative to entropy-based filtering. RSI exhibits drastically reduced threshold sensitivity across model scales.

Figure 4: Left: High-JS token recall by RSI and entropy selection; Right: RSI’s reduced threshold sensitivity, indicating consistently better identification of RL-critical update locations.

Implications and Future Directions

Theoretical: RSI’s derivation as a local coupling of gradient and entropy dynamics positions it as a general bridge between information-theoretic and optimization-centric perspectives on policy learning. This principled approach opens new avenues in understanding token-level credit assignment and the sparsity of effective learning signals.

Practical: RSI-S provides a robust, scale-adaptive method for token curation in RLVR, demonstrating stable and parameter-insensitive performance gains. The entropy-probability coupling also offers diagnostic value: shifts in the empirical RSI distribution may serve as markers for effective RLVR progress or for detecting over-pruning.

Potential future research includes extending RSI-based filtering to non-mathematical domains, leveraging it for credit assignment in multi-reward settings [liu2026gdpo], and investigating its role in enhancing policy generalization and robustness [lu2025generalization, chen2026does]. Token-level selection strategies may also interplay beneficially with emerging RL methods that exploit reasoning chains and compositionality.

Conclusion

The Relative Surprisal Index presents a significant advance in token-level optimization strategies for RLVR, reconciling previously contradictory methodologies under a single, rigorous framework. Its empirical superiority, resilience to parameter settings, and analytic transparency position it as a foundational tool in adaptive token selection for reinforcement learning fine-tuning of LLMs. This paradigm advocates for further exploration of information-theoretically grounded, local policy diagnostics as central drivers of efficient post-training in large-scale LLMs.

---

References:

• "Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index" (2606.31575)
• "Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning" [wang2026beyond]
• "Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs" [yang2025not]
• "Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs" [mengsparse]
• Other referenced works within the essay.