Dopamine-Reward Modeling Pipeline

• Dopamine-Reward Modeling Pipeline is a systematic framework that leverages dopamine-like signals and multi-view sensor data to provide dense, step-aware reward feedback for robotic manipulation.
• It employs a General Reward Model integrating vision and text inputs to generate a discretized progress metric that enhances sample efficiency and robustness in policy learning.
• The pipeline supports one-shot adaptation and policy-invariant reward shaping, enabling rapid domain transfer while preserving optimal policy solutions in reinforcement learning.

The Dopamine-Reward Modeling Pipeline encompasses algorithmic, architectural, and theoretical frameworks that leverage dopamine-like signals for reward assessment and policy learning in high-precision robotic manipulation. This approach centers on the General Reward Model (GRM) and its integration into the Dopamine-RL scheme, employing step-aware reward quantization, multi-perspective fusion, and theoretically sound policy-invariant shaping for reinforcement learning. The resulting pipeline is designed to resolve longstanding challenges in robotic reward design by providing dense, process-level feedback from multimodal sensor data, while provably preserving optimal policy solutions (Tan et al., 29 Dec 2025).

1. Core Pipeline Overview and Mathematical Formalism

The Dopamine-Reward pipeline begins with the acquisition of a task description (natural language) and synchronized multi-view visual observations of the initial state $s_0$, the goal state $s_M$, and pairs of “before” ($s_p$) and “after” ($s_q$) images. The GRM consumes this input and generates a normalized discrete “hop” value $\mathcal H^\star(s_p, s_q) \in [-1, 1]$ representing the relative progress between observed states as a fraction of remaining trajectory distance (Equation 2 in (Tan et al., 29 Dec 2025)).

For a trajectory $\{s_0, s_1, \ldots, s_T\}$, three complementary progress estimates are computed for each timestep $t$:

• Incremental progress: $\Phi_I^\star(s_t)$,
• Forward-anchored progress: $\Phi_F^\star(s_t)$,
• Backward-anchored progress: $\Phi_B^\star(s_t)$.

These are averaged to yield a drift-resistant global progress metric: $$\Phi^\star(s_t) = \frac{1}{3} \left(\Phi_I^\star(s_t) + \Phi_F^\star(s_t) + \Phi_B^\star(s_t)\right)$$ (Eqs. 4–7).

A shaped reward is then defined via: $$r_{\text{shape}}(s_t, s_{t+1}) = \gamma\,\Phi^\star(s_{t+1}) - \Phi^\star(s_t)$$ and combined with a sparse "gold" binary outcome reward $r_{\text{gold}} \in \{0, 1\}$: $$r_{\text{GRM}}(s_t, s_{t+1}) = r_{\text{gold}} + \gamma\,\Phi^\star(s_{t+1}) - \Phi^\star(s_t)$$ (Eq. 22). The shaping term provably does not alter the optimal policy, due to telescoping across the trajectory (Eqs. 25–27).

2. General Reward Model (GRM): Architecture and Training

The GRM leverages a multimodal encoder architecture:

• Vision backbone: Each input image passes through a ViT-style encoder; features are projected via a per-view network to the embedding space of a large decoder-only vision-LLM (Qwen2.5-VL, 3B or 8B).
• Text conditioning: Tokenized task instructions are interleaved with image embeddings.
• Autoregressive prediction: The LLM predicts a discrete hop quantile $\mathcal H^\star$ as a classification over regularly spaced bins ({-100%, ..., 0%, ..., +100%}).
• Training: Supervised with cross-entropy loss on the discrete bins using a 3,400+ hour dataset spanning >100,000 expert trajectories and 350+ tasks. Fine-tuning for new tasks employs an MSE loss over one-shot human demonstrations (Eq. 17).

Key architectural strategies include multi-view input, step-aware discretization, and reward fusion, ensuring high frame-ranking accuracy and robustness to perturbations (Tan et al., 29 Dec 2025).

3. Step-wise Discretization and Multi-Perspective Progress Fusion

To address fine-grained manipulation assessment, process progress is discretized into $N_{\text{hop}}$ uniform bins, mapping continuous hops $\mathcal H$ into a bounded set of tokens. For any pair $(s_p, s_q)$, the ground-truth hop is: $$\mathcal H(s_p, s_q) = \begin{cases} \frac{\Phi(s_q) - \Phi(s_p)}{1 - \Phi(s_p)}, & q \geq p \ \frac{\Phi(s_q) - \Phi(s_p)}{\Phi(s_p)}, & q < p \end{cases}$$ (Eq. 2).

Progress estimation incorporates three calculation anchors:

• $\Phi_I^\star(s_t)$: local increments (by accumulating per-step hops);
• $\Phi_F^\star(s_t)$: progress from the initial state;
• $\Phi_B^\star(s_t)$: progress from the goal (backward perspective).

Averaging these (optionally with consistency-based weights) enables outlier robustness and adapts to drift or distributional shift conditions (Eqs. 4–10).

4. Policy-Invariant Reward Shaping in Dopamine-RL

The Dopamine-RL framework integrates the GRM-predicted process reward into standard reinforcement learning as a potential-based shaping term, as described in the Potential-Based Reward Shaping (PBRS) framework [Ng et al. 1999]. The key theoretical result is: $$r'(s_t, a_t, s_{t+1}) = r_{\text{gold}}(s_t, a_t, s_{t+1}) + \gamma \Phi^\star(s_{t+1}) - \Phi^\star(s_t)$$

• The cumulative shaping reward telescopes to a constant $-\Phi^\star(s_0)$, so the optimal policy is unchanged.
• Any off-the-shelf RL algorithm (e.g., PPO, Cal-QL, ReinFlow) can be used with $r_{\text{GRM}}$ as the reward signal.

This construction fundamentally avoids the “semantic trap” encountered in reward shaping schemes that do not guarantee policy invariance.

5. One-Shot Adaptation and Sample Efficiency

The pipeline supports rapid domain adaptation via “one-shot” supervised fine-tuning of the GRM on a single expert demonstration. The loss minimized is: $$\mathcal L_{\text{SFT}}(\omega) = \mathbb E_{(s_p, s_q) \sim \mathcal D_{\text{human}}} \bigl\| \mathcal H^\star_\omega(s_p, s_q) - \mathcal H_{\text{gt}} \bigr\|_2^2$$ (Eq. 17). This aligns the GRM to the new task or environment in minutes, enabling dense process supervision in real-robot or out-of-distribution conditions.

Empirically, after one-shot adaptation, Dopamine-RL achieves 95% success in only ~150 rollouts (~1 hour of real-robot interaction), a substantial reduction in required data compared to sparse-reward RL or behavioral cloning (Tan et al., 29 Dec 2025).

6. Empirical Evaluations and Ablations

Extensive benchmarking demonstrates:

• GRM accuracy: Multi-view GRM achieves 0.96 video frame rank-correlation (VOC) and 92.8% task success/failure classification, outperforming previous process reward models and single-view baselines.
• Policy learning: Dopamine-RL with shaped rewards improves sample efficiency (e.g., 81% simulation success in 395 vs. 560 rollouts for sparse PPO; 95.2% real-world success in 150 rollouts vs. 68% in 183 for sparse RL).
• Generalization: Robust to out-of-distribution deployment, with 8.3% average drop in performance (vs. 60% drop for behavioral cloning).
• Ablations: Removing multi-perspective fusion (-15–22% SR), policy-invariant shaping (-43.7% SR), or one-shot adaptation (-21.8% SR) each substantially degrade performance.

7. Integration Protocol and Implementation Details

The operational pipeline proceeds as follows:

1. Input preparation: Collect natural-language task instructions and synchronized multi-view images at all key states.
2. GRM adaptation: (Optional) One-shot supervised fine-tuning on new demonstration data.
3. Online rollouts: At each timestep,
   • Observe state via images,
   • Predict hop progress via GRM,
   • Compute and update progress $\Phi^\star(s_t)$,
   • Derive and deliver reward signal $r_{\text{GRM}}$.
4. Policy optimization: Update the agent’s policy by maximizing cumulative shaped reward using any RL algorithm.
5. Evaluation: Assess learning curves, completion success, and reward model reliability.

Training the GRM utilizes 35 million samples from 3,400+ hours of expert and real-world data, with hyperparameters set for large-scale distributed training (e.g., 128×H100 GPUs, ViT and LLM learning rates, AdamW optimizer).

---

Collectively, the Dopamine-Reward Modeling Pipeline provides a unified, scalable, and theoretically grounded approach to reward modeling for robotic RL, incorporating multimodal perception, dense step-aware progress assessment, and policy-invariant reward shaping. Experimental evidence supports its superiority over prior process reward models and sparse-reward approaches in robotic skill acquisition, generalization, and sample efficiency (Tan et al., 29 Dec 2025).