Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training

Abstract: Post-training large foundation models with reinforcement learning typically relies on massive and heterogeneous datasets, making effective curriculum learning both critical and challenging. In this work, we propose ACTOR-CURATOR, a scalable and fully automated curriculum learning framework for reinforcement learning post-training of LLMs. ACTOR-CURATOR learns a neural curator that dynamically selects training problems from large problem banks by directly optimizing for expected policy performance improvement. We formulate problem selection as a non-stationary stochastic bandit problem, derive a principled loss function based on online stochastic mirror descent, and establish regret guarantees under partial feedback. Empirically, ACTOR-CURATOR consistently outperforms uniform sampling and strong curriculum baselines across a wide range of challenging reasoning benchmarks, demonstrating improved training stability and efficiency. Notably, it achieves relative gains of 28.6% on AIME2024 and 30.5% on ARC-1D over the strongest baseline and up to 80% speedup. These results suggest that ACTOR-CURATOR is a powerful and practical approach for scalable LLM post-training.

• The paper presents a novel co-adaptive curriculum framework that formulates data selection as a non-stationary bandit problem to drive RL post-training improvements.
• It employs online stochastic mirror descent and neural function approximation to dynamically select training problems, reducing training steps by up to 80%.
• Empirical evaluations demonstrate significant gains, with improvements of over 28% on challenging benchmarks, confirming the framework’s scalability and robustness.

Actor-Curator: Policy-Improvement Bandits for Co-Adaptive Curriculum Learning in RL Post-Training

Overview and Motivation

The "Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training" (2602.20532) paper introduces Actor-Curator, a fully-automated and scalable framework for curriculum learning in reinforcement learning (RL) post-training of LLMs. Actor-Curator addresses the critical challenge of adaptive data selection from massive, heterogeneous post-training datasets in RL with verifiable reward feedback, focusing on both sample efficiency and final performance in reasoning and mathematics domains.

A central limitation of prior curriculum learning approaches is their inability to generalize in large, non-stationary LLM post-training setups. Existing methods are either reliant on human heuristics, static problem buckets, or tabular per-problem statistics, all of which struggle with scale, dynamic data, and evolving policy distributions during post-training. Actor-Curator fundamentally resolves these limitations by formulating curriculum optimization as a non-stationary stochastic bandit problem, learning an online neural curator that directly optimizes for policy improvement.

Methodology

Actor-Curator adopts a co-adaptive learning paradigm in which an actor (LLM policy) and a curator (problem selector) are trained in a tightly-coupled online loop. The core algorithmic innovation comprises three integrated components:

1. Bandit Formulation of Curriculum: Training problem selection is viewed as a sequential non-stationary bandit, with the curator seeking to maximize the accumulated expected policy improvement. Unlike regression-based or value-prediction approaches, the curator receives bandit feedback measuring first-order policy improvement, computed via differences in expected reward pre- and post-update using importance sampling.
2. Online Stochastic Mirror Descent (OSMD) Optimization: The curator is trained with an OSMD-based update, balancing exploration/exploitation while tracking non-stationary problem utilities. This delivers theoretical dynamic regret guarantees scaling as $O(T^{2/3} V_T^{1/3})$, where $V_T$ quantifies the total variation in arm (problem) utility (policy improvement) sequence.
3. Neural Function Approximation and Scalable Sampling: To operate over large and dynamic datasets, the curator is parameterized as a neural network. A practical two-stage sampling scheme is employed: first, a fixed proposal distribution samples a candidate batch; second, the curator probability-weights the candidate problems. The policy uses a PPO-style proximal clipping surrogate objective to stabilize optimization under neural parameterization.

The resulting training algorithm proceeds in synchronized actor-curator iterations as depicted below.

Figure 1: Actor-Curator’s co-adaptive loop—problem batches are proposed, curated, used for actor RL updates, and feedback is used for curator bandit-based updates via policy improvement estimation.

Empirical Evaluation

Actor-Curator is evaluated on five diverse, challenging benchmarks covering arithmetic reasoning ("Countdown"), logic puzzles ("Zebra"), inductive program induction (ARC-1D), and competition-level math (MATH500, AIME24). Experimental highlights, supported by extensive ablations and head-to-head comparisons against both learning-based (PCL, SEC) and heuristic approaches, include:

• Consistent Outperformance: Across nearly all non-saturated regimes, Actor-Curator achieves higher peak test accuracy within 100 training steps. On hard subsets, gains are pronounced; e.g., +30.5% on ARC-1D and +28.6% on AIME24 over the best baseline.
• Training Efficiency: Actor-Curator achieves target accuracy with up to 80% fewer training steps relative to uniform sampling, indicating a notable improvement in optimization efficiency.

Figure 2: Training accuracy curves on Countdown; Actor-Curator achieves faster convergence and higher final accuracy.

• Robustness to Model and Optimizer: Curator efficacy is demonstrated across multiple actor LLM architectures (Qwen2.5-3B, Llama3.2-3B), and improvements persist for different RL training objectives (e.g., GRPO, GSPO).

Figure 3: Ablation on actor RL algorithm—Actor-Curator consistently outperforms uniform curriculum independent of optimizer choice.

• Scalability: Variations in curator model size and candidate batch size have limited effect on asymptotic performance, with practical stability constraints favoring moderate candidate batch sizes.

  Figure 4: Increasing candidate batch size beyond 2048 leads to instability, validating the need for scalable yet appropriately regularized curation.

• Model-agnostic Curricula: The empirical results on both base and instruction-tuned LLMs further indicate that curriculum benefits are not model-specific but arise from the bandit-driven data selection.

Figure 5: Test accuracy across steps in various benchmarks for Qwen2.5-3B; Actor-Curator surpasses all other sampling strategies consistently.

Design Analysis and Theoretical Properties

Actor-Curator’s regret guarantees are grounded in a rigorous extension of OSMD to the two-stage stochastic action selection setting. The unbiasedness of per-problem policy improvement feedback is formally established under both tabular and function-approximate regimes. The policy-improvement criterion provides a more reliable signal than traditional absolute advantage or value-proxy approaches, especially in highly non-stationary curriculum landscapes.

Ablations indicate that both the OSMD bandit-based objective and policy-improvement learning signal are indispensable for achieving strong curriculum learning. Replacing either—e.g., regressing on absolute advantage—incurs significant performance degradation.

Limitations

Actor-Curator’s quantitative improvements are ultimately bounded by the stability and expressivity of the underlying RL optimizer and the reliability of reward feedback. Noisy or unstable actor updates may persist despite an optimal curriculum. The framework is also predicated on access to meaningful verification-based rewards, which may limit applicability outside math, logic, or code environments. The additional wall-clock cost for curator training (~9-14%) is modest compared to the measured sample efficiency improvements but not negligible in resource-constrained scenarios.

Implications and Future Directions

Actor-Curator provides a robust foundation for curriculum learning in large-scale RL post-training, shifting data selection from static or manually engineered processes to an automated, policy-aware paradigm. This fundamentally improves RL post-training pipelines’ resilience to distribution shift, noise, and ongoing dataset evolution. Actor-Curator’s modular design makes it directly applicable to more complex domains—e.g., multimodal RL, hierarchical reasoning, or lifelong learning—and paves the way for more autonomously adaptive LLM training regimes.

Potential avenues for advancement include integration with search-based RLHF, more expressive curator architectures handling multimodal signals, and closed-loop adaptation to evolving datasets, reward functions, or actors.

Conclusion

Actor-Curator operationalizes co-adaptive curriculum learning via a theoretically-grounded, policy-improvement-based bandit framework. The proposed method delivers consistent performance and efficiency improvements in RL post-training of LLMs, demonstrating that scalable, first-order data curation yields tangible downstream benefits. This approach establishes a principled template for curriculum learning in deep RL settings and highlights the growing importance of automated data selection as a first-class component of foundation model post-training strategies.