Reward Correction in Reinforcement Learning

• Reward correction strategy is a systematic method for adjusting reward signals in sequential decision-making tasks to compensate for noise, bias, and misalignment.
• It employs Bayesian, causal, meta-learning, and kernel-based techniques to recover true objectives and prevent reward hacking in various AI applications.
• Applications include human-robot learning, on-policy distillation, and self-supervised data filtering, enhancing both task accuracy and convergence speed.

A reward correction strategy refers to any systematic method for modifying, adjusting, or interpreting reward signals in sequential decision-making systems, such as reinforcement learning, imitation learning, or human-robot interaction, to compensate for inaccuracies, noise, bias, confounding, or misalignment with the true task or user objectives. Reward correction can be applied at various points: when learning from human feedback, dealing with noisy or misspecified rewards, mitigating reward hacking, or extracting a more reliable optimization signal from automated or learned reward models.

1. Bayesian and Sequence-Based Reward Correction in Human-Robot Learning

Reward correction is central in learning human objectives from physical corrections, particularly when a human provides a sequence of interventions to a robot. In tasks formalized as finite-horizon Markov Decision Processes (MDPs), the robot's baseline reward function $R(\xi; \theta) = \theta \cdot \Phi(\xi)$ (parameterized by weights $\theta$ over features $\Phi(\xi)$) typically diverges from the human's intended reward $R(\xi; \theta^*)$. Physical corrections—timed pushes or pulls—generate a series of trajectory deformations $\{\xi_H^i\}_{i=1}^K$.

To accurately interpret coupled corrections, a joint auxiliary reward function

$$D(\xi_H^1, \dots, \xi_H^K; \theta) = \sum_{i=1}^K \alpha^{K-i} R(\xi_H^i; \theta) - \gamma_h \sum_{i=1}^K \|a_H^i\|^2$$

is used, with $\alpha$ decaying the impact of earlier corrections and $\gamma_h$ penalizing human effort. The robot updates its posterior over $\theta$ online by treating the entire correction sequence as evidence, thereby inferring human objectives more reliably than stepwise or only-final-trajectory approaches. Laplace approximation reduces the computational burden of the partition function. Empirically, reasoning over sequences improves identification accuracy and convergence speed in both simulated and physical robot tasks relative to independent or offline-only correction models (Li et al., 2021).

2. Causal and Preference-Based Reward Correction Against Reward Hacking

Reward correction strategies are critical for mitigating reward hacking—when agents exploit spurious correlations or confounders in proxy reward models. The Causal Reward Adjustment (CRA) technique models reward hacking in external reasoning (for instance, mathematical LLM reasoning chains) as confounding via hidden semantic features $z$. CRA recovers the true causal effect (the "do-intervention" reward) of a reasoning path $\tau$ using Pearl's backdoor adjustment:

$$R_{\mathrm{true}}(\tau) = \sum_{z} P(z) R_{\mathrm{obs}}(\tau \mid z)$$

where $R_{\mathrm{obs}}$ is the PRM-assigned reward. CRA extracts semantic confounders from model activations via sparse autoencoders; it then computes the corrected reward by marginalizing over these features. This strategy demotes logically incorrect, reward-hacked candidates and consistently increases task accuracy (Song et al., 6 Aug 2025).

A complementary strategy, Preference-Based Reward Repair (PBRR), assumes the designer provides a proxy reward $r_p$, possibly misaligned with the ground-truth $r^*$. PBRR iteratively learns an additive correction $\delta(s, a, s'; \theta)$ such that the repaired reward matches human trajectory preferences, focusing only on decreasing reward for transitions that cause misranking. This restricts over-correction and localizes updates, maintaining policy-order invariance. PBRR outperforms methods that learn entire reward functions from scratch and can rapidly eliminate reward hacking with far fewer human queries (Hatgis-Kessell et al., 14 Oct 2025).

3. Meta-Learning-Based and Bi-level Reward Correction in Process Reward Models

For intermediate stepwise rewards in complex reasoning (e.g., code generation, mathematical proofs), MC estimates of correctness at each step are often highly noisy. Both DreamPRM-Code and FunPRM employ meta-learning-based correction mechanisms to purify these partial rewards.

The core idea is to co-optimize model weights and (noisy) labels by using a clean meta-objective derived from final program test outcomes. For example, DreamPRM-Code treats the intermediate labels as learnable, using a bi-level setup: (1) update PRM parameters on current (possibly corrected) labels, (2) perturb labels to minimize meta-loss measured on a small set of clean final-solution data. FunPRM operationalizes this correction as a lightweight, instance-level table applied to MC-estimated labels. In both, the reward model is explicitly regularized to produce intermediate rewards that are maximally predictive of reliable final outcomes, yielding consistent improvements in final task performance metrics such as pass@1 on LiveCodeBench (Zhang et al., 17 Dec 2025, Zhang et al., 29 Jan 2026).

4. Correcting Reward Signal Bias, Noise, and Confounders

A distinct class of reward correction targets statistical noise and bias induced by external verifiers, shortcut exploitation, or spurious features:

• Noisy Verifiers: In RLVR with imperfect binary verifiers, observed rewards $\tilde R$ may bear deterministic false positive/negative rates $(\rho_0, \rho_1)$. Unbiased policy gradient estimation is restored either by backward correction (solving an affine system for surrogate rewards) or forward correction (adjusting score-function weights based on known noise rates). An LLM-based appeals mechanism provides online $\rho_1$ estimation for real-world FN-dominated verifiers (Cai et al., 1 Oct 2025).
• Kernel-Based Shortcut Mitigation: PRISM constrains a learned reward model to be invariant to predefined shortcut group transformations (e.g., verbosity, sycophancy) by incorporating Haar-integrated invariant kernels and batch-level global decorrelation terms in the learning objective. This regularization suppresses spuriously correlated shortcut features in both reward prediction and subsequent policy optimization (Ye et al., 21 Oct 2025).
• Dual reward and adaptive shaping: Techniques like DeepCompress apply an adaptive length-based reward correction that dynamically penalizes long reasoning chains for "simple" problems and rewards exploratory depth for "hard" problems. This prevents reward hacking via length and balances efficiency with accuracy (Liang et al., 31 Oct 2025).

5. Harmonizing Coarse and Fine-Grained Reward Models

Blending granular process and outcome reward models can introduce misleading gradients or reward hacking. The PRocess cOnsistency Filter (PROF) avoids composite rewards and instead filters trajectories for RL updates by keeping only (a) correct responses with high process reward or (b) incorrect responses with low process reward, maintaining training stability and improving both final accuracy and intermediate reasoning quality. This filtering is shown to outperform naive reward mixing and reduces step-wise value estimation noise (Ye et al., 3 Sep 2025).

6. Reward Correction in On-Policy Distillation and Data Selection

In dense KL-constrained RL or on-policy distillation (OPD), reward correction can enhance student policy learning. The G-OPD/ExOPD framework introduces a tunable reward scaling factor, shown to enable "reward extrapolation"—the student can surpass the teacher by emphasizing the RL reward (teacher–reference log ratio) more heavily than the KL penalty. Further, for strong-to-weak distillation, using the teacher's pre-RL base model as the reference for reward computation sharpens the signal and further enhances the student's achievable performance boundary (Yang et al., 12 Feb 2026).

A direct application of reward correction to self-supervised data filtering is demonstrated by using model-internal confidence scores as a proxy for reward, shown in CRew and CRew-DPO, which consistently outperforms various trained reward models for LLM self-training by leveraging token-level predictive uncertainty and confidence gap-driven preference pair generation (Du et al., 15 Oct 2025).

7. Design Considerations, Practical Guidelines, and Limitations

Across domains, a successful reward correction strategy hinges on:

• Faithfully modeling coupled evidence (as in trajectory-sequence or group-invariant schemes) rather than treating observations as i.i.d.
• Avoiding bias/variance trade-offs by matching correction strength (e.g., penalty on human effort, shortcut invariance) to task and observation properties.
• Efficient implementation: precomputing normalizers or correction tables, parallelizing feature extraction, or adopting lightweight bi-level meta-learners.
• Having access to small, reliable meta-values (unit tests, trusted human feedback) for anchoring correction.
• Recognizing limitations, such as the need for explicit shortcut feature identification or the assumption of stationary or class-conditional noise models. Unmodeled confounders or mis-specified correction structure remain sources of residual bias.

Reward correction is thus an expansive, toolkit-like concept encompassing theoretical (e.g., Bayesian, causal, kernel-invariant), algorithmic (online filtering, meta-learning, backdoor adjustment), and practical (domain-specific, proxy-repair, self-training) strategies, each tailored to the application’s signal properties and failure modes.