LP-ACRL: Learning Progress-based Curriculum RL

• LP-ACRL is a framework that leverages real-time learning progress to automatically adjust task difficulty, ensuring efficient skill acquisition.
• The method tracks episodic rewards and computes performance differentials, using a softmax update rule to adapt the task-sampling distribution.
• Empirical results show significant speedups, robust performance, and effective transfer in domains such as quadruped locomotion.

A Learning Progress-based Automatic Curriculum Reinforcement Learning (LP-ACRL) framework adaptively generates curricula for RL agents by dynamically prioritizing tasks or subtasks where the agent’s performance is improving most rapidly. Rather than relying on static curriculum sequences or manually specified difficulty orderings, LP-ACRL quantifies the agent’s learning progress as an intrinsic signal and reweights the task-sampling distribution in response, allowing for scalable, data-driven curriculum generation in both discrete and combinatorially large task spaces (Fournier et al., 2018, Kanitscheider et al., 2021, Li et al., 24 Jan 2026). This paradigm has been shown to substantially accelerate skill acquisition, improve robustness against forgetting, and enable high-performance deployment in complex real-world domains such as rough-terrain quadruped locomotion (Li et al., 24 Jan 2026).

1. Formalism and Core Algorithmic Principles

LP-ACRL is instantiated over an agent’s RL environment $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma)$$ and a discrete (often multidimensional) set of parameterized tasks $\mathcal{T} = \{\zeta\}$. Each curriculum stage $j$ utilizes a task-sampling distribution $c_j \in \Delta(\mathcal{T})$ to draw tasks for agent rollouts. The fundamental LP-ACRL loop consists of:

1. Episodic Reward Tracking: For each task $\zeta$, the empirical average episodic return is computed under $c_j$ as $R_{c_j}(\zeta)$, typically by averaging over the last $M$ trajectories.
2. Learning Progress Estimation: The one-step learning progress is $$LP_{c_j}(\zeta) = R_{c_j}(\zeta) - R_{c_{j-1}}(\zeta)$$, reflecting changes in task-specific competence.
3. Adaptive Sampling Update: The next curriculum distribution is updated via a softmax over learning progress:

$$c_{j+1}(\zeta) = \frac{\exp(LP_{c_j}(\zeta)/\beta)}{\sum_{\zeta'}\exp(LP_{c_j}(\zeta')/\beta)}$$

where $\beta$ is a temperature parameter controlling exploration.

1. Policy Update: The agent’s parameters are updated using all data collected within the current curriculum stage.

This mechanism forms a closed feedback loop in which the curriculum tracks the "frontier" of the agent's competence, biasing sampling toward those tasks currently yielding the steepest improvements and, by construction, gradually advancing to harder or more diverse regions of the task space (Li et al., 24 Jan 2026).

2. Learning Progress Metrics, Variants, and Extensions

Learning progress can be estimated in various ways, depending on domain and data availability:

• Reward Differential: The canonical LP metric is the difference in mean episodic reward across consecutive windows, $LP(\zeta)=R_{new}(\zeta)-R_{old}(\zeta)$ (Li et al., 24 Jan 2026).
• Success Probability Slope: In goal-based or binary success tasks, LP can be the linear-regression slope over recent success rates (Kanitscheider et al., 2021).
• Competence Progress: Absolute finite-difference of sliding windowed test accuracies for each difficulty parameter (e.g., target accuracy $\epsilon$ in reacher tasks) (Fournier et al., 2018).
• Gradient-norm Based: Teacher rewards based on the mean or summed per-step policy gradient norm, capturing parameter-space adaptation directly (Campbell et al., 2023).
• Complexity-gain: Variational complexity gain, reflecting the increase in model description length, and prediction-gain, i.e., the reduction in model loss, for supervised or RL paradigms (Graves et al., 2017).

Variants such as bidirectional measures (tracking both progress and regress to prevent forgetting), mastering-rate extensions (see (Willems et al., 2020)), or zone-of-proximal-development motivated objectives (see ProCuRL (Tzannetos et al., 2023)) further refine the curriculum-generation logic by explicitly accounting for task dependency structure or optimal task difficulty.

3. Integration into RL Architectures and Algorithms

LP-ACRL can be instantiated in a variety of RL architectures:

• Policy Gradient and Actor-Critic: Integration with DDPG/UVFA (Fournier et al., 2018), PPO (Kanitscheider et al., 2021), or IMPALA-style recurrent architectures allows for curricula over high-dimensional and partially observable tasks.
• Teacher-Student MDP: The curriculum itself is cast as a teacher MDP, with the “teacher” selecting task parameters for the “student”, and the teacher’s reward is the observed LP metric (Campbell et al., 2023).
• Meta-RL / CMDP Formulations: Curriculum sequencing is formalized as a higher-level Markov Decision Process whose state encodes the agent's knowledge and whose actions correspond to task selection; optimal curriculum policies are learned by nested RL schemes (Narvekar et al., 2018).
• Replay-based Variants: Sampling mechanisms that resemble Prioritized Level Replay, but with priorities tied to online learning progress rather than fixed TD error (Li et al., 24 Jan 2026).

Empirical implementations maintain moving windows/frame buffers of performance data for online LP estimation and employ softmax, proportional, or thresholded sampling to realize the curriculum schedule (Kanitscheider et al., 2021).

4. Empirical Results and Real-World Impact

In a range of domains, including OpenAI-Gym Reacher (Fournier et al., 2018), visual Minecraft skill acquisition (Kanitscheider et al., 2021), dialogue policy induction (Zhao et al., 2020), and high-DoF quadrupedal locomotion (Li et al., 24 Jan 2026), LP-ACRL consistently yields:

• Significant reductions in sample complexity (e.g., $2.5\times$–$3\times$ speedup over hand-crafted or uniform curricula).
• Higher asymptotic returns and zero-shot deployment performance.
• Increased robustness against catastrophic forgetting, especially when employing bidirectional LP estimation.
• Automatic adaptation to diverse or unstructured task spaces, with no need for manual task ordering or handcrafted progression heuristics.
• Effective scaling to hundreds of interleaved or compositional skills (e.g., $600$-task rough terrain pools for ANYmal locomotion).

Notably, (Li et al., 24 Jan 2026) demonstrates real-world transfer by distilling LP-ACRL policies into a student deployed at 2.5 m/s across all terrain types, exceeding prior curriculum baselines.

5. Limitations, Challenges, and Theoretical Considerations

While LP-ACRL eliminates the need for static curricula, it is not without drawbacks:

• Sensitivity to Curriculum Granularity: Discretization of continuous task spaces may cause combinatorial task explosion (Li et al., 24 Jan 2026).
• Hyperparameter Tuning: The choice of temperature $\beta$, LP window size, and reward normalization affects stability and may require domain-specific tuning (Fournier et al., 2018, Li et al., 24 Jan 2026).
• Delayed or Sparse Progress: LP may be zero for extended periods in extremely challenging or unsolvable regions, reducing adaptation.
• No Strong Convergence Guarantees: No theoretical results guarantee optimality or sample complexity bounds; existing analyses are empirical (Li et al., 24 Jan 2026).
• Task Interdependence: LP sampling may be suboptimal when task mastery requires strict prerequisite structure, though extensions such as Mastering-Rate (Willems et al., 2020) or ProCuRL (Tzannetos et al., 2023) address this.

Table: Comparison of Select Implementations

Domain	LP Metric	Curriculum Update	Notable Features	Reference
Quadruped RL	Reward diff.	Softmax over LP	600 tasks; real-world transfer	(Li et al., 24 Jan 2026)
Visual RL	EMA success-prob	Sigmoid/std. LP	107 tasks; exploration bonus	(Kanitscheider et al., 2021)
Dialogue RL	Reward delta	DQN-based teacher	Over-repetition penalty	(Zhao et al., 2020)

6. Extensions and Related Directions

Recent work has extended LP-ACRL in multiple dimensions:

• Gradient-based LP: Directly employing parameter update magnitude as a progress signal to address the reward-barren or poorly shaped domains (Campbell et al., 2023).
• Task Dependency: Mastering-rate and DAG-based attention mechanisms to avoid over-sampling mastered or impossible tasks and efficiently sequence prerequisites (Willems et al., 2020).
• Safety-Critical Curriculum: Combining LP-ACRL with constraint-satisfying interventions and probabilistic guarantees for safe exploration (Turchetta et al., 2020).
• Meta-Curriculum Learning: Learning not just curricula, but the curriculum-generation algorithm itself, via Bayesian optimization or outer-loop RL (Turchetta et al., 2020).
• Proximal Development: ZPD-based criteria (ProCuRL) for selecting tasks at the optimal "learning edge," with mathematical linkages to LP maximization and robust empirical performance (Tzannetos et al., 2023).

A plausible implication is that LP-ACRL will serve as the foundational paradigm for scaling RL to highly diverse, real-world, open-ended task spaces, provided limitations surrounding task discretization, curriculum stability, and cross-task transfer are addressed in future work.

7. Summary

Learning Progress-based Automatic Curriculum Reinforcement Learning leverages principled, online estimates of an agent's learning progress to drive adaptive, data-efficient, and robust curriculum generation in RL. By continuously tracking competence improvements and adjusting task-sampling distributions, LP-ACRL systems are able to automatically sequence training, efficiently acquire complex skill sets, and maintain strong performance even as the task portfolio grows in scale and heterogeneity (Li et al., 24 Jan 2026, Kanitscheider et al., 2021, Fournier et al., 2018, Campbell et al., 2023, Tzannetos et al., 2023). Continuing advances in LP metrics, dependency modeling, and task-space generalization are expanding the scope and effectiveness of this paradigm.