TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics

Abstract: While Vision-Language-Action (VLA) models have seen rapid progress in pretraining, their advancement in Reinforcement Learning (RL) remains hampered by low sample efficiency and sparse rewards in real-world settings. Developing generalizable process reward models is essential for providing the fine-grained feedback necessary to bridge this gap, yet existing temporal value functions often fail to generalize beyond their training domains. We introduce TOPReward, a novel, probabilistically grounded temporal value function that leverages the latent world knowledge of pretrained video Vision-LLMs (VLMs) to estimate robotic task progress. Unlike prior methods that prompt VLMs to directly output progress values, which are prone to numerical misrepresentation, TOPReward extracts task progress directly from the VLM's internal token logits. In zero-shot evaluations across 130+ distinct real-world tasks and multiple robot platforms (e.g., Franka, YAM, SO-100/101), TOPReward achieves 0.947 mean Value-Order Correlation (VOC) on Qwen3-VL, dramatically outperforming the state-of-the-art GVL baseline which achieves near-zero correlation on the same open-source model. We further demonstrate that TOPReward serves as a versatile tool for downstream applications, including success detection and reward-aligned behavior cloning.

• The paper introduces TOPReward, a training-free framework that extracts dense, temporal reward signals from VLM token probabilities.
• The method reframes progress estimation by computing log-probabilities of an affirmative token, achieving strong VOC and binary success rates on benchmarks.
• TOPReward demonstrates robust zero-shot performance across diverse manipulation tasks, enabling improved policy optimization in RL and imitation learning.

TOPReward: Token Probabilities as Zero-Shot Hidden Rewards for Robotic Progress Estimation

Problem Motivation and Background

Real-world deployment of Vision-Language-Action (VLA) models faces substantial barriers stemming from sample inefficiency endemic to sparse reward regimes in Reinforcement Learning (RL). Contemporary reward modeling approaches are typically either domain-specific (requiring expensive manual annotation or task-specific demonstrations) or lack robust generalization, especially across embodiments and visual domains. While recent efforts investigated leveraging pretrained Vision-LLMs (VLMs) for zero-shot reward estimation, these approaches often depend on querying models for text-based progress scores—a protocol highly sensitive to the unreliability of instruction following and numeric output calibration in open-source VLMs.

The paper posits that pretrained VLMs, despite being dismissed as insufficient for robotics reward modeling, have latent progress understanding capabilities not evident in generated text, but manifest in their internal token-level probability distributions.

Method: Token Probability-Based Reward Modeling

TOPReward is introduced as a probablistically grounded framework for extracting temporal value functions from the internal token probability distributions of pretrained video VLMs. It discards the unreliable process of prompting VLMs for scalar/numeric progress outputs and instead reframes progress estimation as measuring the temporal evolution of model confidence in trajectory-instruction completion.

Given a sequence of observation frames $\tau_{1:T}$ and a language instruction, TOPReward computes for each time prefix the conditional log-probability (from the VLM) of a special affirmative token (e.g., "True") given the prompt "Does this trajectory complete the task?". This approach constructs a trajectory-aligned, temporally dense progress curve without any additional model fine-tuning.

The per-trajectory progress signal is then normalized via per-episode min-max scaling, resulting in a continuous signal in $[0, 1]$. This signal is both temporally consistent and interpretable as a value function for downstream imitation and RL applications.

Figure 1: TOPReward enables zero-shot estimation of progress across diverse manipulation tasks, delivering temporally consistent visual reward signals suitable for success detection, policy improvement, and benchmark evaluation.

Empirical Evaluation

Large-Scale Real-World Progress Estimation

TOPReward is evaluated on two major benchmarks:

• Open X-Embodiment (OXE): 39 datasets × 20 episodes each (spanning robot platforms and camera configurations).
• Mani: A new benchmark introduced in the paper comprising 130+ manipulation tasks across Franka, YAM (single/bimanual), and SO-100/101 with fine-grained stage annotations.

On both benchmarks, progress orderings are compared using the Value-Order Correlation (VOC) metric, quantifying monotonic progress alignment.

Strong results are observed in zero-shot settings, with Qwen3-VL-8B achieving mean VOC 0.857 on OXE and 0.947 on Mani, outperforming GVL by substantial margins on open-source models (GVL yields near-zero or negative VOC). Notably, GVL exhibits high VOC only on proprietary VLMs, confirming TOPReward's advantage when restricted to widely available open-source backbones.

Figure 2: Mean dataset-level VOC for GVL and TOPReward across OXE and Mani evaluation sets, with TOPReward consistently outperforming GVL on open-source models.

In detailed trace analysis, TOPReward yields monotonically increasing, smooth progress curves that closely follow ground-truth subtask boundaries, robust to visual and temporal variation. Gemini-GVL (when available) is less stable, with frequent fluctuations and spurious non-monotonicity.

Figure 3: Example progress traces: TOPReward (orange) aligns with ground-truth subtask boundaries, outperforming Gemini-GVL (blue) in consistency and monotonicity.

Success Detection

A documented failure mode for VOC-based approaches is their insensitivity to failed trajectories that plateau at sub-maximal progress (high VOC can still be achieved without task completion). TOPReward's probability-based signal yields a more discriminative metric—average final-step log-probability of "True"—directly reflecting the predicted satisfaction of the specified instruction.

In binary success/failure classification (ROC-AUC metric) on Mani, TOPReward achieves 0.654 ROC-AUC on Qwen3-VL-8B, compared with 0.519 for GVL (random performance). On Gemini, both models are near parity, demonstrating that TOPReward resolves key limitations in GVL, especially for open-source VLMs.

Figure 4: VOC's inability to distinguish completed from incomplete plateauing trajectories can lead to high scores on fundamentally failed executions.

Figure 5: "True" token probability exhibits the sharpest distinction between successful and failed trajectories, motivating its use as the completion indicator.

Advantage-Weighted Behavior Cloning

TOPReward supplies dense, temporally aligned rewards suitable for weighting demonstrations in offline imitation learning. In experiments on 6 SO-100 real-world tasks, advantage-weighted behavior cloning (AWR) policies fine-tuned with TOPReward progress signals consistently outperform standard behavior cloning (BC), e.g., achieving 10/10 successful completions in difficult settings where BC achieves only 7 or fewer.

Figure 6: Overview of the six real-world single-arm SO-100 manipulation tasks for AWR evaluation.

Figure 7: For the task "Place doll in box," only the AWR policy fine-tuned using TOPReward achieves consistent success, outperforming pretrained and BC baselines.

Analysis and Ablations

Comprehensive ablation reveals that prompt formatting (e.g., chat templates) significantly degrades progress signal quality, undermining VOC by as much as 50% on Qwen3-VL-8B. This sensitivity underscores the importance of aligning reward probing mechanisms with the model’s pretraining objectives and inherent next-token prediction protocol.

Implications and Future Directions

TOPReward demonstrates that open-source VLMs, even without domain adaptation or fine-tuning, have emergent capabilities for temporally fine-grained reward modeling in complex real-world robotic tasks. By mining token-level affirmations from pretrained video-language priors, TOPReward exposes a previously overlooked avenue for dense reward specification, crucial for scalable RL and data curation in robotics.

Practically, this enables:

• Automated ranking, filtering, and labeling of demonstration data across varied tasks and robots.
• Zero-shot deployment for success detection and progress monitoring.
• Direct integration into offline RL and AWR pipelines, yielding improved policy sample efficiency and generalization.

Theoretically, the findings suggest a decoupling between a VLM’s instruction-following/numeric reasoning abilities and its internal multimodal state representation—a critical point for future architectural and training dataset design.

Limitations remain: ultimate performance is bottlenecked by the perceptual acuity of the underlying VLM, and min-max normalization is currently per-trajectory. Nevertheless, as open-source VLMs mature (especially in video understanding), the presented methodology stands to gain predictive fidelity.

Conclusion

TOPReward introduces a robust, training-free pipeline for extracting temporally dense, generalizable reward signals from pretrained VLMs through token probability analysis. It achieves strong empirical performance on large-scale benchmarks, surpasses prior state-of-the-art in zero-shot progress estimation, and enables new capabilities for automatic dataset curation and policy optimization in robotics. While susceptible to limitations imposed by VLM perception, the methodology provides a foundation for advancing scalable reward modeling as open-source video-LLMs continue to improve.