IB-GRPO: Pareto-Optimal Learning Path Framework

• IB-GRPO is a multi-objective learning path recommendation framework that uses vector-valued rewards to optimize pedagogical objectives such as learning effect and ZPD alignment.
• The framework integrates genetic algorithms and offline RL for expert warm-start, followed by indicator-based policy optimization without manual scalarization.
• Empirical evaluations on ASSIST09 and Junyi datasets demonstrate its superior balance in optimizing learning effect, path diversity, and operational constraints.

IB-GRPO (Indicator-Based Group Relative Policy Optimization) is a learning path recommendation (LPR) framework that aligns LLM-based policies with pedagogical objectives such as learning effect maximization, Zone of Proximal Development (ZPD) alignment, operational constraints, and path diversity, by leveraging a vector-valued reward structure and direct Pareto frontier optimization without manual scalarization (Wang et al., 21 Jan 2026).

1. Motivation and Pedagogical Objectives

Long-horizon LPR requires generating sequences of learning items personalized to individual students in order to:

• Maximize long-term learning effect: Enhance a student’s post-instruction proficiency.
• Schedule exercise difficulty to match the ZPD: Adjust item difficulty to maintain challenge within a beneficial proficiency band.
• Respect operational constraints: Adhere to desired path lengths and session constraints.
• Maintain diversity: Avoid repetitive recommendations by supporting a wide range of plausible learning trajectories.

Standard LLMs, trained to optimize next-token likelihood, exhibit myopic planning and do not natively conform to these multi-faceted pedagogical needs. Simple scalar reduction of multi-objective signals risks “locking in” suboptimal trade-offs. IB-GRPO introduces a vector reward formulation and directly targets the Pareto-optimal frontier, sidestepping ad hoc weighting and promoting adaptive balancing of objectives.

2. Formal Framework and Multi-Objective MDP

The LPR setting is formalized as an episodic Markov Decision Process (MDP):

• State ($s$): Encapsulates the student's latent proficiency ($a$), interaction history $H=\{(c_1, y_1),\dots,(c_k, y_k)\}$, and prompt features (e.g., target path length, prior recommendations).
• Action ($\pi_t$): Choosing the next knowledge concept or exercise from a candidate set.
• Trajectory ($\pi$): An ordered sequence $(\pi_1,\dots,\pi_L)$ generated auto-regressively by the parameterized LLM policy $\mu_\theta$.

The reward function $r(s, \pi) \in \mathbb{R}^4$ is a vector comprising:

1. Learning Effect $E_p(\pi)$:

$E_p(\pi) = (E_e - E_s)/(E_{sup} - E_s)$

where $a$0 are pre/post-test scores and $a$1 the maximum.

2. ZPD Alignment $a$2:

$a$3

$a$4 is the item difficulty; $a$5 the optimal-difficulty center for proficiency $a$6.

3. Length Constraint $a$7:

$a$8

where $a$9.

4. Diversity $H=\{(c_1, y_1),\dots,(c_k, y_k)\}$0:

$H=\{(c_1, y_1),\dots,(c_k, y_k)\}$1

$H=\{(c_1, y_1),\dots,(c_k, y_k)\}$2 is the sampled path group, and similarity is measured over $H=\{(c_1, y_1),\dots,(c_k, y_k)\}$3-gram overlaps.

The learning objective is to train $H=\{(c_1, y_1),\dots,(c_k, y_k)\}$4 so that non-dominated trajectories sampled from the policy closely approximate the true Pareto frontier of the vectorized reward, without scalar collapse.

3. IB-GRPO Algorithmic Procedure

IB-GRPO operates in two distinct stages: a hybrid expert warm-start via supervised fine-tuning (SFT), followed by indicator-based group relative policy optimization.

3.1 Stage I: Hybrid Expert Warm-Start

• Genetic Algorithm Expert: Each trajectory is encoded as a chromosome; tournament selection, crossover, and mutation drive exploration of the learning effect reward with diversity.
• Policy-based Teacher (Offline RL): Pre-trained LPR agents (e.g., CSEAL, GEPKSD) complement genetic search, especially in sparse or low-performing regions.
• Behavior Cloning SFT: Expert trajectories from both sources comprise dataset $H=\{(c_1, y_1),\dots,(c_k, y_k)\}$5. The LLM is warm-started to maximize the log-likelihood:

$H=\{(c_1, y_1),\dots,(c_k, y_k)\}$6

• Outcome: Warm-started policy $H=\{(c_1, y_1),\dots,(c_k, y_k)\}$7.

3.2 Stage II: Indicator-Based Group Relative Policy Optimization

Training Iteration Procedure:

1. Group Sampling: For each state in the batch, sample $H=\{(c_1, y_1),\dots,(c_k, y_k)\}$8 trajectories $H=\{(c_1, y_1),\dots,(c_k, y_k)\}$9 using the current policy.
2. Reward Vectorization: Compute reward vector $\pi_t$0 for each trajectory.
3. $\pi_t$1 Dominance Indicator: For pairs $\pi_t$2,

$\pi_t$3

quantifies the minimum uniformly needed augmentation for $\pi_t$4 to weakly dominate $\pi_t$5.

1. Pareto Fitness $\pi_t$6:

$\pi_t$7

with scaling parameter $\pi_t$8.

2. Group-Relative Advantage $\pi_t$9: Standardize $\pi$0 within the group

$\pi$1

3. Policy Update: Optimize using importance sampling and asymmetric clipping

$\pi$2

$\pi$3, with $\pi$4, $\pi$5.

This pairwise dominance approach facilitates direct discovery of Pareto-optimal policies and circumvents the drawbacks of manual scalarization.

4. Training Workflow and Pseudocode

The core training protocol can be summarized as follows:

Stage	Key Steps
Warm-start SFT	Generate expert trajectories (GA, RL agents); Aggregate dataset $\pi$6; Fine-tune LLM via cross-entropy loss on experts.
IB-GRPO Optimization	For $\pi$7 epochs: sample state batches; for each state, sample $\pi$8 trajectories, compute vector rewards, all pairwise $\pi$9, $(\pi_1,\dots,\pi_L)$0, $(\pi_1,\dots,\pi_L)$1; update $(\pi_1,\dots,\pi_L)$2 on $(\pi_1,\dots,\pi_L)$3; update reference policy.

This approach synthesizes search-based and offline RL-generated data for broad solution coverage, then proceeds with indicator-based policy optimization.

5. Empirical Evaluation

Experiments are conducted on the ASSIST09 and Junyi datasets with a DKT-based Knowledge Evolution Simulator (KES). Key experimental characteristics:

• Datasets:
  • ASSIST09: 167 concepts, 4,217 learners, 346,860 records.
  • Junyi: 835 concepts, 525,061 learners, 21,460,249 records.
• Simulator: KES (DKT environment) for counterfactual student response simulation.
• Model backbone: Qwen2.5-7B LLM (8 × A800-40GB GPUs).
• Key hyperparameters: Group size $(\pi_1,\dots,\pi_L)$4, indicator scale $(\pi_1,\dots,\pi_L)$5 tuned per validation, clipping $(\pi_1,\dots,\pi_L)$6.

Baselines:

• Non-RL: DKTRec
• General RL: DQN, Actor-Critic, PPO
• Education-specific RL: CSEAL, GEPKSD
• LLM-based: GenAL, ReAL

Metrics:

• Learning Effect ($(\pi_1,\dots,\pi_L)$7): (higher is better)
• LenScore: $(\pi_1,\dots,\pi_L)$8
• Path Diversity ($(\pi_1,\dots,\pi_L)$9): $\mu_\theta$0
• ZPD alignment ($\mu_\theta$1)

Main Results:

Dataset	Baseline (L=20)	IB-GRPO (L=20)
Junyi	ReAL: 0.5724	0.7743
ASSIST09	GEPKSD: 0.5837	0.5911

IB-GRPO surpasses all baselines across metrics and path lengths. Ablations reveal that removal of ZPD rewards degrades long-horizon planning, and alternatives to $\mu_\theta$2 such as HVO or GDPO underperform in achieving balanced trade-offs. Hybrid GA+RL demonstration data enables the most comprehensive Pareto coverage.

Diagnostic plots confirm that IB-GRPO trajectories are more tightly distributed around the ZPD optimal band and exhibit reduced late-stage difficulty variance. Achieved solutions are superior in balancing learning effect, ZPD compliance, path length accuracy, and diversity.

6. Significance and Extensions

IB-GRPO demonstrates that direct Pareto-efficient policy optimization for LPR, leveraging indicator-guided group advantages and pedagogical alignment, yields substantial improvement over both generic reinforcement learning and simple LLM-based approaches. By explicitly incorporating a differentiable ZPD reward and circumventing fixed scalarization, IB-GRPO promotes more robust and pedagogically sound learning path recommendations that generalize across datasets and path horizons. This suggests that indicator-based group relative methods offer a scalable foundation for multi-objective alignment in complex educational inference tasks.

For a detailed description of the framework, see (Wang et al., 21 Jan 2026).