Min-Form Credit Assignment

• Min-form credit assignment is a method that replaces aggregated sum-form rewards with minimal, structure-guided feedback for critical decision steps.
• It employs algorithms like PURE and modular RL to control reward hacking, enhance stability, and reduce gradient variance in lengthy action sequences.
• Empirical results demonstrate up to a 9.8% LLM fine-tuning improvement and faster transfer rates, showing its practical benefits in complex RL settings.

Min-form credit assignment refers to a family of credit assignment strategies in optimization and reinforcement learning (RL) that minimize the propagation of redundant or uncontrolled feedback signals when assigning credit for long-term outcomes to individual actions, tokens, or components in a decision sequence. Unlike canonical sum-form credit assignment, which aggregates (discounted) cumulative rewards, min-form schemes apply minimal or structure-guided feedback, often via local, modular, or sparse mechanisms. These approaches have been developed and analyzed to address challenges such as reward hacking, instability, inefficient transfer, and lack of interpretability in RL, stochastic computation graphs, and supervised or biologically inspired learning systems.

1. Theoretical Foundations and Formal Definitions

Min-form credit assignment is defined by its departure from traditional RL value functions and gradient estimators, focusing instead on minimum-sufficiency or modularity principles.

In standard RL, the typical sum-form value function aggregates discounted future rewards: $$Q^{\mathrm{sum}}_\pi(s_t,a_t) = \mathbb{E}\Biggl[\sum_{k=0}^{\infty}\gamma^k\,r_{t+k}\Bigm|s_t,a_t\Biggr]$$ Min-form credit assignment replaces this with operations such as the minimum of a trajectory's process rewards: $$G(s_t, a_t|\tau) = \min\{r^p_t, \dots, r^p_n\}, \qquad w = \arg\min_{i\le n}r^p_i$$ so that only steps at or before the "worst" event receive nonzero credit, and steps beyond are assigned zero. The value function becomes

$$Q^{\mathrm{min}}_\pi(s_t,a_t) = \mathbb{E}_\tau[G(s_t, a_t|\tau)]$$

At a more abstract level, as in modular RL, min-form credit assignment is characterized by the independence of feedback signals: $$I_\text{alg}(\delta_1, ..., \delta_T \mid x, \theta) \simeq 0$$ where $\delta_t$ is the feedback signal (e.g. gradient, TD-error) for mechanism $t$, $x$ is the trajectory, and $\theta$ are the model parameters. This ensures that credit assigned to one component does not redundantly leak into others (Chang et al., 2021).

In structured computation graphs, the "min-form" estimator refers to the minimal-variance, locally bootstrapped critic constructed by systematically conditioning on as much of the non-descendant graph (parents or ancestors) as feasible, reducing the total sampling variance and computational overhead (Weber et al., 2019).

2. Key Methodological Variants

Several recently introduced min-form or minimal-form credit assignment mechanisms exemplify these principles across different learning settings.

a) PURE and Min-Form with Process Reward Models

The PURE algorithm (Cheng et al., 21 Apr 2025) reformulates RL fine-tuning of LLMs with process reward models (PRMs), replacing canonical discounted-sum credit assignment with the minimum over step rewards. Given trajectory rewards $r^p_i$, a soft-min weighting

$$r_i^{p*} = \frac{\exp(-r^p_i / T)}{\sum_{j=1}^n \exp(-r^p_j / T)}\, r^p_i$$

approximates hard minimum assignment as $T\to 0^+$. This approach mitigates reward hacking by narrowing the value function's range, stabilizing training, and distributing step-wise feedback such that all steps up to the first error receive the same "worst-case" credit (Cheng et al., 21 Apr 2025).

b) Modular Credit Assignment via Algorithmic Independence

Modular RL (Chang et al., 2021) formalizes min-form credit assignment as the enforcement of algorithmic independence among per-decision feedback signals. The modularity criterion is tested using algorithmic causal graphs (ACML), where only single-step TD (TD(0)) with separate parameterizations for each action ensure true independence of updates (no shared path dependencies). Policy-gradient and multi-step TD methods violate this independence by construction.

c) Minimal-Variance Gradient Estimation in SCGs

In stochastic computation graphs, the min-form estimator relies on constructing critics and baselines using the largest Markov conditioning set of non-descendants of each stochastic node. Truncated bootstrapping (partial evaluation) ensures only minimal local feedback is used, reducing sample variance without introducing bias (Weber et al., 2019).

d) Minimized Control in Biologically Plausible Systems

In structured neural systems, minimizing control (rather than output loss) as a learning objective yields "min-form" local plasticity: the magnitude of required top-down control feedback is penalized, leading to credit assignment rules that use only locally available voltages and signals. This yields fully local, biologically plausible update rules that are robust to noise and require no global coordination (Meulemans et al., 2022).

3. Motivations and Problems Addressed

Min-form credit assignment strategies are motivated by specific limitations of canonical and correlated credit signals:

• Reward Hacking: In dense process reward settings, e.g., PRMs for LLM reasoning steps, discounted-sum credit assignment is highly susceptible to pathological behavior—models may exploit repeated "thinking" steps or generate long outputs with high-priority tokens to inflate cumulative reward (Cheng et al., 21 Apr 2025).
• Training Instabilities: Sum-form credit assignment causes rapid training collapse with process rewards, as observed with standard PPO or RLHF. Min-form avoids this by constraining value function growth and distributing blame or credit more conservatively (Cheng et al., 21 Apr 2025).
• Transfer and Unlearning: Non-modular, correlated feedback (as in policy gradients or multi-step TD) hinders transfer: an update due to a late-sequence change may "unlearn" unrelated early sequence decisions. Modular/min-form variants enable faster adaptation in transfer scenarios (Chang et al., 2021).
• Sample Efficiency and Variance: Classical global returns or improper baselines induce high gradient variance. Locally bootstrapped critics and exploitation of Markov structure via min-form estimation reduce variance and computation (Weber et al., 2019).
• Biological Plausibility: Infinitesimal feedback in gradient-based learning is unrealistic; min-form strategies such as minimizing control or energy cost offer strong, local, and robust credit assignment compatible with observed cortical dynamics (Meulemans et al., 2022).

4. Algorithms and Implementation Details

Min-form credit assignment is realized in practice through methodological choices in RL and optimization loops. Below, representative algorithms are summarized.

Algorithm/System	Key Min-Form Mechanism	Credit Assignment Principle
PURE (Cheng et al., 21 Apr 2025)	Process reward soft-minimum	Value function = min-future
Modular RL (Chang et al., 2021)	TD(0), action-wise separation	Independence of δ's
Min-form SCG (Weber et al., 2019)	Critic: largest possible conditioning	Local, partial bootstrapping
Min-control (DFC) (Meulemans et al., 2022)	Minimize steady-state feedback magnitude	Surrogate control objective

Concretely, the PURE algorithm transforms dense process rewards using softmin, then computes PPO advantages over the transformed rewards. All PPO machinery remains unchanged, but the aggregation of step rewards uses the minimal or soft-min form only. Empirical evidence demonstrates that sum-form assignment with process rewards leads to training collapse, whereas min-form matches performance of verifiable-reward methods in ∼30% of the training steps (Cheng et al., 21 Apr 2025).

Modular RL is implemented by decomposing policy into separately parameterized per-action networks, and using TD(0) with no shared hidden variables to guarantee modular gradient updates. In the empirical regime, modular min-form methods exhibit 3-14× faster transfer in sparse intervention tasks relative to policy-gradient baselines (Chang et al., 2021).

In neuro-inspired systems, the min-form objective becomes minimizing the control/feedback effort required to achieve target outputs, leading to local, online updates for forward and feedback synapses, with empirical noise robustness and performance competitive with backpropagation (Meulemans et al., 2022).

5. Empirical Results and Applications

Substantial experimental evaluation demonstrates the effectiveness of min-form credit assignment.

• LLM Reasoning Fine-tuning: On benchmarks including MATH-500, Minerva, OlympiadBench, AIME24, and AMC23, PURE with min-form achieves +9.8% average improvement over base models, matches verifiable reward RLHF within 30% of training steps, and is robust to reward hacking (Cheng et al., 21 Apr 2025).
• Transfer Efficiency: Modular TD(0) methods show 3–14× faster adaptation on transfer tasks in discrete MDPs, especially when task changes occur late in long decision sequences. Non-modular algorithms suffer severe unlearning (Chang et al., 2021).
• Variance and Computation: In SCGs, min-form estimation enables efficient gradient computation by leveraging partial bootstrapping, avoiding the need to evaluate full subtrees or entire descendant graphs (Weber et al., 2019).
• Biological and Artificial Systems: Minimizing control in deep feedback networks results in performance close to backpropagation (e.g., 2.19% MNIST error versus 1.83% for backprop), with pronounced robustness to noise and single-phase, local plasticity rules (Meulemans et al., 2022).

6. Limitations, Open Questions, and Future Directions

Despite substantial advances, min-form credit assignment has inherent limitations and unresolved challenges.

• Residual Reward Hacking: Min-form cannot fully eliminate spurious positive signals under flawed or non-fluent process rewards. Cases include repetitive completions, single-step completions, or trivial outputs that game the reward model. Supplementing with sparse verifiable rewards partially mitigates this (Cheng et al., 21 Apr 2025).
• Expressivity and Flexibility: Strict modularity may limit learning capacity in deeply entangled or cyclical tasks. Modular RL is proven only for acyclic decision sequences; extensions to cyclic or highly interdependent settings are unresolved (Chang et al., 2021).
• Critic and Baseline Construction: The effectiveness of min-form estimators relies on optimal Markov/conditioning set choices for critics and baselines; practical approximations (e.g. via bootstrapped value networks) may trade variance for bias (Weber et al., 2019).
• Biological Plausibility vs. Efficiency: Min-control objectives in neural networks increase biological plausibility but may yield slower convergence in some practical settings compared to global optimization (Meulemans et al., 2022).
• Future Directions: Proposed remedies include iterative co-training of generative process reward models with LLMs, adaptive or mixed credit assignment strategies, and investigation of min-form assignment in generative, non-Markovian, or highly modular contexts (Cheng et al., 21 Apr 2025).

7. Relation to Classical and Contemporary Approaches

Min-form credit assignment generalizes or departs sharply from classical RL and gradient optimization frameworks. The approach unifies several algorithmic innovations:

• λ-returns and eligibility traces: Min-form mechanisms, such as those in GRPO-λ, rely on algebraic equivalence between GAE-style eligibility traces and per-token decaying memory, enabling fine-grained, critic-free credit propagation for LLM fine-tuning (Parthasarathi et al., 30 Sep 2025).
• Actor-Critic and Bootstrapped Gradients: Min-form gradient estimation in SCGs encompasses actor-critic, TD(λ), and partial average methods as special cases, casting standard algorithms within a broader unifying framework (Weber et al., 2019).
• Single-phase, local learning: In biological learning, min-form differentiates from traditional backprop or multi-phase algorithms by enforcing locality and robustness, accommodating empirical neural constraints (Meulemans et al., 2022).

The min-form perspective thus acts as both a theoretical and practical framework, driving improved stability, modularity, and interpretability in credit assignment across a spectrum of artificial and biological systems.