Curriculum-Based Training Paradigm

• Curriculum-Based Training Paradigm is a methodology that structures learning sequentially from simpler to more challenging tasks to enhance optimization.
• It employs a Curriculum Markov Decision Process (CMDP) framework to automatically select and transfer knowledge between tasks.
• Empirical studies in Gridworld and Ms. Pac-Man demonstrate significant training step reductions (up to 50%) with robust transfer across tasks.

A curriculum-based training paradigm is a general methodology in machine learning and reinforcement learning wherein the learning process is structured as a progression from simpler to more complex tasks, training examples, or environment conditions. The underlying principle is to facilitate more efficient optimization and improved generalization by first mastering easier subproblems and then leveraging the acquired knowledge or representations when facing harder challenges. This paradigm encompasses a wide range of strategies, from manual task sequencing to fully automatic curriculum-policy learning framed as reinforcement learning within a meta-MDP. Curriculum-based approaches offer both practical empirical benefits and strong theoretical motivation as formalized in both supervised and RL domains.

1. Foundations and Motivations

Curriculum learning in RL is formally defined as employing a sequence of source MDPs, $M_1, M_2, \dots, M_{t-1}$, culminating in a hard target MDP $M_t$. The agent trains sequentially on each $M_i$, transferring learned knowledge with the goal of reducing sample complexity and improving asymptotic and converged performance on the target. This is particularly relevant for environments with sparse rewards or complex dynamics, where exploration and value function convergence are otherwise prohibitively slow. By exploiting transferable structures such as value functions, options, or potential-based reward shaping, a curriculum helps the agent converge faster, achieve more robust exploration (reducing catastrophic failures), and often acquire superior policies in complex domains (Narvekar et al., 2018).

The motivation for curriculum learning is grounded in its ability to smooth non-convex optimization landscapes, denoise supervision signals, and serve as an instance of continuation methods, as examined in broader CL surveys (Wang et al., 2020). Formal theory and empirical ablations confirm that curricula often accelerate convergence and improve generalization bounds relative to direct end-to-end training on the hardest task.

2. Curriculum Markov Decision Process (CMDP) Framework

The process of automatically learning curricula in RL is cast as a meta-MDP, termed the curriculum MDP (CMDP):

$$M^C = \bigl(\mathcal{S}^C, \mathcal{A}^C, P^C, R^C, S_0^C, S_f^C\bigr)$$

• $\mathcal{S}^C$ (curriculum states): Encodes the current knowledge state. If the agent’s policy or value function uses parameters $\bm{\theta}$, $s^C = \bm{\theta}$. In reward shaping settings, $s^C$ is the vector of accumulated potential weights.
• $\mathcal{A}^C$ (curriculum actions): Each action is a source MDP $M_i$; the CMDP action at a step is "train agent on $M_i$".
• $P^C(s^C, a^C, s'^{C})$: Probability that, after training on $a^C$ from $s^C$, the agent reaches parameterization $s'^{C}$.
• $R^C(s^C, a^C)$: Negative of the training cost (environment steps or wall-clock time) when training $a^C$ from $s^C$.
• $S_0^C$: Initial state (randomly initialized or untrained parameters).
• $S_f^C$: Terminal set (any $\bm\theta$ achieving specified threshold $\delta$ on $M_t$).

A curriculum policy $\pi^C: \mathcal{S}^C \to \mathcal{A}^C$ prescribes which source task to train on at each knowledge state. Solving for the optimal policy $\pi^{C*}$ yields the curriculum that minimizes total training cost to reach a desired target proficiency (Narvekar et al., 2018).

3. Learning and Representing Curriculum Policies

State and Feature Representation

• Value-transfer setting: For a linear value function $Q_{\bm\theta}(s,a) = \bm\theta \cdot \bm\phi(s,a)$, CMDP state is $\bm\theta$. One may reconstruct a full Q-table or directly tile-code each entry.
• Reward shaping: CMDP state is the sum of transferred potential weights; features are computed via tile-coding the joint weight vector.

Policy Approximation and Training

CMDP action-value $$Q^C_{\bm w}(s^C, a^C) \approx \bm w \cdot \bm\phi^C(s^C, a^C)$$ is learned via standard RL algorithms (e.g., Sarsa($\lambda$), $\epsilon$-greedy). Each CMDP episode corresponds to executing the current curriculum policy until the agent reaches target proficiency, accumulating (negative) cost. The learned weights $\bm w$ define the deterministic curriculum policy $\pi^C(s^C) = \arg\max_{a^C} Q^C_{\bm w}(s^C, a^C)$.

Adapting to Multiple Transfer Algorithms

Transfer mechanism affects only the feature representation of CMDP states and the transition dynamics $P^C$. The overall CMDP formulation and learning protocol remain unchanged regardless of whether transfer is via value functions or potential-based reward shaping.

4. Empirical Evaluation and Results

Empirical studies demonstrate substantial speed-ups and reliability gains from curriculum-based RL:

• Gridworld: 10 source tasks + complex target. Learned curricula via CMDP reduced training steps by 40–50% relative to no curriculum, matching or slightly exceeding manually crafted curricula. Both discrete and continuous CMDP representations converge after a few hundred episodes.
• Ms. Pac-Man: 15 source mazes + complex target. Value-function transfer curricula reduced training by >50% over no curriculum, and reward-shaping curricula achieved an extra 20–30% improvement in convergence speed. The CMDP method is robust to stopping criteria for source tasks—curricula can optimally learn to revisit tasks (Narvekar et al., 2018).

In both domains, learned curriculum policies adaptively selected the most informative next training task and automatically accommodated different transfer mechanisms.

5. Extensions and Theoretical Implications

Curriculum-based training can be situated within the broader CL literature as an explicit formalization of automated curriculum design (Wang et al., 2020). In the CMDP approach, curriculum learning is operationalized as a planning/optimization problem, where off-the-shelf RL methods are deployed at the meta-level. This results in several implications:

• Policy robustness: Meta-learned curricula adapt to the specific learning dynamics and transfer characteristics of the base agent, often outperforming fixed or heuristic schedules.
• Transfer across mechanisms: The CMDP abstraction unifies curricula for both value-based and potential-based transfer, hinting at extensibility to further transfer paradigms.
• Scalability: By encoding the state as the agent’s parameterization and leveraging function approximation, the approach is poised to scale to high-dimensional or continuous agent representations.
• Automatic curriculum discovery: The CMDP-based paradigm requires no expert knowledge to design curricula, with policies emerging from data as the agent interacts with source tasks.

6. Best Practices and Practical Lessons

• Parameterization: Careful design of CMDP state representations (e.g., effective feature extraction from agent parameters) is essential to policy performance.
• Agent transfer interfaces: The CMDP approach presupposes the ability to extract, represent, and transfer parameters or potentials between source and target MDPs.
• Termination criteria: The policy's robustness to source-task stopping conditions enables flexible and efficient curricula even with coarse or ad-hoc stopping mechanisms.
• Empirical tuning: While meta-level RL brings generality, empirical hyperparameter search (e.g., step size for Sarsa, tile-coding grids) often remains necessary in practice.

7. Context in Broader Curriculum Learning Taxonomy

This paradigm typifies the trend toward automatic curriculum design, moving away from static, hand-designed curricula toward data-driven policies optimized for rapid and robust acquisition of nontrivial tasks. It incorporates and extends the principles articulated in foundational CL surveys—namely, the difficulty-measurer and scheduler framework—by embedding them in an explicit optimization and policy-learning setting (Wang et al., 2020). CMDP-based curriculum discovery complements other approaches such as teacher-student bandit or RL–teacher systems, further establishing curriculum-based RL as a central methodology for tackling sparse-reward, multi-task, or transfer settings.