Progressive GRPO Training Paradigm

• Progressive GRPO training paradigm is a reinforcement learning framework that uses structured, staged updates and competence-aware rewards to overcome issues like reward saturation and policy–reward mismatch.
• It integrates curriculum strategies and composite rewards (e.g., visual fidelity, temporal coherence, semantic alignment) to adaptively refine policy performance across diverse generative tasks.
• Progressive GRPO techniques optimize computational efficiency and sample diversity via dynamic trajectory selection, data synthesis, and scaffolded hinting, achieving significant performance gains.

A progressive GRPO training paradigm refers to reinforcement learning schemes within the Group Relative Policy Optimization (GRPO) framework that employ structured, staged, or competence-aware strategies for updating the policy, reward models, or data distribution. These paradigms are designed to overcome key challenges such as reward saturation, distributional bias, accumulation of uninformative trajectories, and misaligned learning signals, particularly as model capabilities evolve during training. Progressive GRPO techniques have been widely applied across video generation, LLM reasoning, multimodal data synthesis, and sample-efficient generative modeling.

1. Motivations and Conceptual Foundations

Traditional GRPO employs a frozen, pre-trained reward model to score generated samples, with the policy updated via a group-relative advantage function that compares each rollout’s reward to its group mean or variance. This approach exhibits fundamental limitations:

• Reward Saturation and Policy–Reward Mismatch: As the generator exceeds the reward model’s capacity, reward signals saturate, leading to vanishing gradients and impaired learning. Early-stage training, conversely, over-penalizes low-quality outputs, destabilizing policy improvement and introducing semantic biases (Li et al., 24 Nov 2025).
• Computational Inefficiency and Reward Clustering: Large group sizes, essential for stable policy gradients, exacerbate computational costs and cause many trajectories to cluster near the mean, providing minimal optimization value (&&&1&&&).
• Exploration Stalling (the “Learning Cliff”): When the policy encounters problems beyond its ability (e.g., mathematically complex questions), all trajectories receive zero reward, nullifying learning signals and stalling progress (Zhang et al., 22 Oct 2025).
• Limited Data Diversity: Static datasets and training batches constrain the scope of model exploration, reducing reinforcement-driven discovery or reasoning skill acquisition (Huang et al., 24 Nov 2025).

Progressive GRPO paradigms introduce structural advances—curriculum schedules, competence-aware reward blending, scaffolded hinting for hard cases, data synthesis pipelines, or adaptive trajectory selection—all tailored to align learning signals with evolving policy capabilities.

2. Competence-Aware and Self-Paced Reward Schedules

The self-paced GRPO paradigm introduces a composite, competence-dependent reward whose terms shift in relative emphasis according to the generator’s measured ability:

• Composite Reward Decomposition: The total reward $R_t(\tau)$ is a weighted sum:

$$R_t(\tau) = w_{fid}(c_t) R_{fid}(\tau) + w_{temp}(c_t) R_{temp}(\tau) + w_{sem}(c_t) R_{sem}(\tau)$$

where $R_{fid}$ assesses visual fidelity, $R_{temp}$ assesses temporal coherence, and $R_{sem}$ quantifies text–video semantic alignment (Li et al., 24 Nov 2025).

• Competence Scalar $c_t$: A monotonic scalar $c_t \in [0,1]$ incremented over training (typically $c_{t+1} = \min(1, c_t + \Delta c)$), orchestrates the transition from coarse-grained to fine-grained objectives.
• Smooth Weight Transitions: Weights $w_j(c_t)$ are implemented via a temperature-controlled softmax over transition scores, e.g., $g_{fid}(c) = 1-c$, $g_{temp}(c)=c(1-c)$, $g_{sem}(c) = c$, with normalization ensuring $\sum_j w_j(c_t) = 1$.
• Clipped Surrogate Objective: Policy updates are performed under the composite reward using a normalized advantage and PPO-like clipping:

$$A^{(t)}_i = \frac{R_t(\tau_i) - \mu_t}{\sigma_t}, \quad J^{SP}(\theta) = \mathbb{E}_{i,t}\left[\min(\rho_{t,i} A^{(t)}_i, \mathrm{clip}(\rho_{t,i}, 1-\epsilon, 1+\epsilon)A^{(t)}_i )\right]$$

(Li et al., 24 Nov 2025).

• Curricular Effects: The progressive reward mitigates reward exploitation and bias, prevents early saturation, and fosters improved semantic alignment and temporal coherence, as evidenced by VBench and VideoAlign metrics (Li et al., 24 Nov 2025).

3. Expand-and-Prune and Trajectory Diversity Maximization

Proactive GRPO (“Pro-GRPO”) employs a dynamic strategy to address reward clustering and computational overhead in generative modeling:

• Reward Clustering Phenomenon: Standard GRPO groups often yield many trajectories with rewards close to the group mean, contributing little to the gradient and wasting computation (Ge et al., 17 Dec 2025).
• Optimal Variance Filtering (OVF): OVF selects a subset of rollouts maximizing within-subset reward variance, leading to stronger learning signals; empirically, $k \approx G/2$ is effective for subset size.
• Expand-and-Prune Pipeline: The initial group $G_{max}$ is expanded, and at multiple internal checkpoints, trajectories are pruned based on fast proxy rewards computed from latent features. Only survivors proceed to completion, and the PPO-style update is computed using this diverse, high-variance survivor set (Ge et al., 17 Dec 2025).
• Computational Advantages: Expand-and-prune delivers significant FLOPs reductions (up to 41%) with improved or preserved in-domain and out-of-domain sample quality, as measured by PickScore, HPSv2.1, and ImageReward benchmarks (Ge et al., 17 Dec 2025).

4. Scaffolded Hinting and Adaptive Curriculum for Reasoning Tasks

Scaf-GRPO addresses the “learning cliff” in LLM reasoning by progressively injecting hierarchical hints only when the policy becomes stagnant on “true-hard” prompts:

• Stagnation Diagnosis: Batch-level zero-reward rate $\Phi(t)$ is tracked; learning is deemed stagnant if $$|\Phi(t) - \Phi(t-\Delta t)| < \tau_{\mathrm{plateau}}$$ persists (Zhang et al., 22 Oct 2025).
• Multi-Tiered Hint Hierarchy: Hints are injected in three tiers (Knowledge $\rightarrow$ Planning $\rightarrow$ Solution), each decomposable into incremental pieces. The injection is conservative: only the minimum needed hint is introduced, and only one rollout per problem group is augmented, ensuring learning remains on-policy.
• Loss Integration: The group-relative PPO loss accommodates rollouts with (conditional on hint presence) different prompts, maintaining valid policy gradients.
• Empirical Gains: On Qwen2.5-Math-7B, pass@1 rises from 30.0% (vanilla GRPO) to 43.3% (Scaf-GRPO) on AIME24, with +9.2% average improvement over prefix-continuation methods and effective out-of-domain generalization (Zhang et al., 22 Oct 2025).
• Ablation Results: Removal of hierarchy, incremental hinting, or the initial exemption period diminishes gains, affirming the necessity of the progressive structure.

5. Online Data Synthesis and Self-Evolving Datasets

Syn-GRPO interleaves GRPO updates with asynchronous, model-instructed data synthesis:

• Dual-Loop Structure: GRPO rollouts include chain-of-thought reasoning, new image descriptions, and diversity self-estimates. The highest-diversity description is asynchronously sent to a data server, which synthesizes new tasks by controlled outpainting, ensuring foreground consistency and label validity (Huang et al., 24 Nov 2025).
• Diversity Reward: A smoothed, batch-wise diversity metric $\mathcal{V}(q)$ guides both policy improvement and sample selection, with the rollout diversity reward given by $$R_{diversity}(o_g) = 1 - |v_g - \tilde{\mathcal{V}}(q)|$$ after exponential moving average smoothing.
• Policy–Environment Feedback: As the dataset evolves, the model encounters ever richer, self-generated data. This directly increases the exploration scope and generalization capacity over standard GRPO or static entropy-regularized baselines.
• Empirical Performance: On RefCOCO, OVD, and ISR benchmarks, Syn-GRPO outperforms original GRPO by 4–10 mAP/accuracy points, with ablations confirming sustained performance improvements as the dataset grows (Huang et al., 24 Nov 2025).

6. Multi-Stage and Curriculum Strategies for Multimodal/Temporal Reasoning

ReasonAct demonstrates a progressive GRPO training paradigm for fine-grained video reasoning:

• Three-Stage Pipeline: (1) Foundational text-only reasoning, (2) video-based supervised chain-of-thought (CoT) fine-tuning, and (3) reinforcement learning with a temporally consistent, sub-action aware GRPO reward (Liu et al., 3 Aug 2025).
• Reward Decomposition: The final episode reward combines task correctness, biomechanically motivated sub-action coverage, and temporal consistency over reasoning traces, yielding advantages for PPO updates. Temporal consistency is evaluated via metrics such as sequence order (Kendall’s $\tau$), entity tracking, and temporal binding.
• Cumulative Efficacy: Each stage builds on the last; performance increments are maintained at each transition. For Qwen2.5-VL-3B, the progressive pipeline yields up to +17.9% (HMDB51), +15.8% (UCF-101), +12.3% (Kinetics-400) over non-progressive baselines (Liu et al., 3 Aug 2025).

7. Empirical Impact and Measured Benefits

Progressive GRPO methods yield improvements across a broad spectrum of quantitative and qualitative axes:

Approach	Key Metric Improvement	Characteristic Effect
Self-Paced GRPO (Li et al., 24 Nov 2025)	+0.76 VBench (Wan2.1-1.3B), +0.63 (14B)	Reduces reward/policy mismatch, semantic bias
Scaf-GRPO (Zhang et al., 22 Oct 2025)	+44.3% pass@1 (AIME24), +12.6% (7 average)	Restores learning on “hard” examples
Pro-GRPO (Ge et al., 17 Dec 2025)	1.26–1.41× FLOP savings, higher PickScore	FLOPs efficiency and sampling diversity
Syn-GRPO (Huang et al., 24 Nov 2025)	+4–10 accuracy/mAP points	Self-curated, diverse, ever-improving datasets
ReasonAct (Liu et al., 3 Aug 2025)	+17.9% accuracy (HMDB51)	Enables small models to approach large-LM video scores

Ablation and stability results underscore the importance of curriculum schedule, dynamic reward composition, and staged learning for robust generalization and convergence (Li et al., 24 Nov 2025, Zhang et al., 22 Oct 2025, Ge et al., 17 Dec 2025, Huang et al., 24 Nov 2025, Liu et al., 3 Aug 2025).

• "Growing with the Generator: Self-paced GRPO for Video Generation" (Li et al., 24 Nov 2025)
• "Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning" (Zhang et al., 22 Oct 2025)
• "Expand and Prune: Maximizing Trajectory Diversity for Effective GRPO in Generative Models" (Ge et al., 17 Dec 2025)
• "Syn-GRPO: Self-Evolving Data Synthesis for MLLM Perception Reasoning" (Huang et al., 24 Nov 2025)
• "ReasonAct: Progressive Training for Fine-Grained Video Reasoning in Small Models" (Liu et al., 3 Aug 2025)