LaSeR: Reinforcement Learning with Last-Token Self-Rewarding

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a core paradigm for enhancing the reasoning capabilities of LLMs. To address the lack of verification signals at test time, prior studies incorporate the training of model's self-verification capability into the standard RLVR process, thereby unifying reasoning and verification capabilities within a single LLM. However, previous practice requires the LLM to sequentially generate solutions and self-verifications using two separate prompt templates, which significantly reduces efficiency. In this work, we theoretically reveal that the closed-form solution to the RL objective of self-verification can be reduced to a remarkably simple form: the true reasoning reward of a solution is equal to its last-token self-rewarding score, which is computed as the difference between the policy model's next-token log-probability assigned to any pre-specified token at the solution's last token and a pre-calculated constant, scaled by the KL coefficient. Based on this insight, we propose LaSeR (Reinforcement Learning with Last-Token Self-Rewarding), an algorithm that simply augments the original RLVR loss with a MSE loss that aligns the last-token self-rewarding scores with verifier-based reasoning rewards, jointly optimizing the reasoning and self-rewarding capabilities of LLMs. The optimized self-rewarding scores can be utilized in both training and testing to enhance model performance. Notably, our algorithm derives these scores from the predicted next-token probability distribution of the last token immediately after generation, incurring only the minimal extra cost of one additional token inference. Experiments show that our method not only improves the model's reasoning performance but also equips it with remarkable self-rewarding capability, thereby boosting its inference-time scaling performance.

• The paper introduces LaSeR, a framework that converts reasoning verification into a last-token self-rewarding score for efficient reinforcement learning.
• It leverages a closed-form reward, computed from next-token log probabilities, to eliminate costly external verifiers and reduce inference overhead.
• Empirical results show enhanced reasoning accuracy and self-verification F1 scores nearing 80% across various LLM architectures and benchmarks.

LaSeR: Reinforcement Learning with Last-Token Self-Rewarding

Introduction and Motivation

The LaSeR framework introduces a principled and efficient approach to jointly optimize reasoning and self-verification capabilities in LLMs under the Reinforcement Learning with Verifiable Rewards (RLVR) paradigm. The core challenge addressed is the lack of reliable verification signals at inference time, where ground-truth answers are unavailable. Prior approaches either rely on external verifiers—incurring significant computational and engineering overhead—or require the LLM to generate both solutions and self-verifications sequentially, which doubles inference cost and reduces efficiency. LaSeR circumvents these inefficiencies by leveraging a theoretical insight: the true reasoning reward can be equivalently represented by a last-token self-rewarding score, computed from the model’s own next-token log-probabilities.

Theoretical Foundation and Algorithmic Design

LaSeR’s central theoretical result is that the RL objective for self-verification admits a closed-form solution: the verifier-based reward for a solution is equal to the scaled log-probability ratio between the policy and reference models for a pre-specified token at the last position of the generated solution. This insight enables the definition of the last-token self-rewarding score:

$$r_s = \beta_v \log \frac{\pi_\theta(z_c|\mathbf{x}, \mathbf{y})}{\pi_{\text{ref}}(z_c|\mathbf{x}, \mathbf{y})}$$

where $z_c$ is a special token (e.g., an unused token), $\pi_\theta$ is the policy model, $\pi_{\text{ref}}$ is the reference model, and $\beta_v$ is the KL regularization coefficient.

Empirically, $\log \pi_{\text{ref}}(z_c|\mathbf{x}, \mathbf{y})$ is nearly constant across all $(\mathbf{x}, \mathbf{y})$ pairs, allowing it to be precomputed as $c_{\text{ref}}$. This reduces the self-rewarding score to a function of the policy model’s output and a constant, eliminating the need for reference model inference during training and inference.

The LaSeR algorithm augments the standard RLVR loss with a mean squared error (MSE) loss that aligns the last-token self-rewarding score with the verifier-based reward:

$$L = \mathbb{E}_{\mathbf{x}, \mathbf{y}} \left( r_s - r_v(\mathbf{x}, \mathbf{y}) \right)^2$$

This joint loss enables simultaneous optimization of both reasoning and self-verification capabilities, with minimal additional computational cost—only one extra token inference per sample at test time.

Implementation Details and Practical Considerations

• Token Selection: $z_c$ should be an unused special token to ensure $\pi_{\text{ref}}(z_c|\mathbf{x}, \mathbf{y})$ is minimal and stable.
• Reference Log-Probability Simplification: Precompute $c_{\text{ref}}$ on a small sample set; this halves the computational cost of self-rewarding score calculation.
• Class-Level Loss Re-Weighting: To address class imbalance between correct and incorrect solutions, apply dynamic re-weighting within each batch, ensuring balanced self-verification performance.
• Advantage Integration: Combine verifier-based and self-rewarding-based advantages during RL optimization to mitigate misjudgments by rule-based verifiers and provide finer-grained reward signals.
• Warm-Up Phases: Sequentially warm up reasoning and self-rewarding capabilities to stabilize training, especially when initializing from base models.
• Inference-Time Usage: At test time, the self-rewarding score can be used for solution ranking, weighted majority voting, or as a confidence measure for self-verification.

Empirical Results

LaSeR demonstrates consistent improvements in both reasoning accuracy and self-verification F1 scores across multiple LLM architectures (LLaMA, Qwen) and mathematical reasoning benchmarks (MATH500, AMC23, AIME24/25, OlympiadBench). Notably, LaSeR-trained models achieve self-verification F1 scores approaching or exceeding 80%, outperforming equally sized state-of-the-art external verifiers and matching the performance of much larger reward models.

Figure 1: The majority voting (Maj@K) and weighted majority voting (RM@K) results on MATH500 and OlympiadBench.

Weighted majority voting using the last-token self-rewarding scores (RM@K) yields superior inference-time scaling compared to unweighted majority voting (Maj@K), as shown in Figure 1. This demonstrates the practical utility of the self-rewarding signal for aggregating multiple solutions and improving final answer accuracy.

Figure 2: The generalizability of LaSeR on general reasoning tasks.

LaSeR’s generalizability to non-mathematical reasoning tasks is also evaluated. While the self-rewarding signal is less discriminative in general domains (e.g., MMLU-Pro, GPQA-Diamond), it still provides useful information for weighted voting, and does not degrade the model’s core reasoning ability.

Figure 3: Cumulative implicit reward values across 32 reasoning trajectories sampled from Open-Reasoner-Zero-7B on an AIME2024 problem. Red lines correspond to wrong solutions and green lines correspond to correct solutions.

Figure 3 illustrates the length bias in implicit reward: longer (often incorrect) solutions accumulate higher implicit rewards, making them unreliable for solution ranking. The last-token self-rewarding score, being length-invariant, avoids this pitfall.

Figure 4: The curves of training rewards and training self-verification F1 scores under different combinations of hyper-parameters with EMA smoothing (EMA coef.=0.9).

Figure 5: The curves of training rewards and training self-verification F1 scores of our method with and without class-level loss re-weighting practice (EMA coef.=0.9).

Figure 6: The comparison of the training dynamics between the last-token self-rewarding loss and the SFT loss.

Figures 5–7 provide ablation analyses, showing that:

• The self-rewarding MSE loss does not interfere with reasoning optimization, unlike supervised fine-tuning (SFT) losses, which can degrade reasoning performance.
• Class-level loss re-weighting is essential for balanced self-verification.
• The choice of hyperparameters ($\beta_v$, $\alpha$) affects the trade-off between reasoning and self-verification, but LaSeR is robust within reasonable ranges.

Comparison with Prior Work

LaSeR contrasts with previous self-verification approaches that require separate solution and verification generations, incurring double the inference cost. By extracting the self-rewarding signal from the last token’s next-token distribution, LaSeR achieves near-zero additional cost and seamless integration into existing RLVR pipelines. Unlike implicit reward-based ranking, which is length-biased, LaSeR’s last-token score is length-invariant and theoretically grounded.

Limitations and Future Directions

• General Reasoning Tasks: The self-rewarding signal is less effective in domains where the model or verifier is weak, suggesting a need for improved verifiers or domain adaptation.
• Further Cost Reduction: The authors propose eliminating even the single extra token inference by leveraging the log-probability at the <EOS> position, though this may require careful handling of generation termination.
• Enhanced Self-Rewarding: Increasing the number of additional inference tokens for self-rewarding could further improve calibration, at the expense of efficiency.
• Broader RL Algorithms: While LaSeR is demonstrated with GRPO, its integration with other RL algorithms (e.g., PPO, DAPO, VAPO) remains to be explored.

Conclusion

LaSeR provides a theoretically principled and practically efficient method for equipping LLMs with robust self-verification capabilities, tightly coupled with their reasoning optimization. By reducing the self-verification objective to a last-token self-rewarding score, LaSeR enables high-accuracy, low-cost self-verification and improved inference-time scaling. The approach is broadly applicable to RLVR-based LLM training and sets a new standard for efficient, unified reasoning and verification in large-scale LLMs.