RL Forgets! Towards Continual Policy Optimization

Abstract: Continual post-training is becoming a central paradigm for adapting vision-LLMs to evolving tasks. Recent work has increasingly favored reinforcement learning over supervised fine-tuning, driven by the belief that reinforcement learning is inherently less prone to forgetting. However, the belief remains insufficiently validated, as existing evidence is largely drawn from outdated or homogeneous benchmarks. To revisit this assumption, we introduce MRCL, a Multimodal Reasoning Continual Learning benchmark built from diverse and recently released multimodal datasets. Experiments on MRCL show that reinforcement learning can still suffer from severe catastrophic forgetting during continual post-training. To address this challenge, we propose Continual Policy Optimization (CPO), a replay-free framework grounded in the prior-task behavioral KL objective. CPO uses a theoretically justified parameter-movement regularization to limit policy drift on previous tasks. Extensive experiments across multiple model scales demonstrate that CPO consistently reduces forgetting while preserving, and in some cases improving, pretrained model capabilities. On Qwen3-VL-8B, CPO reduces forgetting by 13.7\% and improves pretrained capability by 7.0\%. The implementation code is available at https://github.com/MaolinLuo/CPO.

• The paper introduces Continual Policy Optimization (CPO) to address catastrophic forgetting in reinforcement learning for VLMs.
• It utilizes a sparse, masked L1 regularization scheme based on parameter movement proxies to maintain prior task performance.
• Empirical results on the MRCL benchmark demonstrated significant improvements over classical RL and SFT methods in task retention.

RL Forgets! Towards Continual Policy Optimization: A Technical Analysis

Motivation and Problem Statement

The widespread deployment of vision-LLMs (VLMs) in open-ended, continual learning environments necessitates robust methods for adaptation to evolving data streams without catastrophic forgetting. While recent literature asserts that reinforcement learning (RL)-based post-training confers greater resistance to forgetting compared to supervised fine-tuning (SFT), these claims predominantly rely on outdated, overlapping, or insufficiently challenging benchmarks. The foundational question addressed in this work is whether RL is intrinsically robust against forgetting during continual post-training in VLMs, especially when evaluated under realistic, diverse, and up-to-date task distributions.

MRCL Benchmark: Realistic Multimodal Continual Learning

This paper introduces MRCL—a Multimodal Reasoning Continual Learning benchmark—constructed from multimodal datasets released after mid-2025. MRCL is designed to:

• Minimize contamination from pretraining corpora, ensuring evaluation rigor.
• Span a range of cognitive reasoning tasks (e.g., medical VQA, navigation, visual-spatial puzzles, mathematical reasoning, chart and document understanding).
• Challenge both System 1 and System 2 capabilities, probing compositional reasoning, spatial analysis, and out-of-distribution generalization.

MRCL imposes a sequential learning paradigm representative of real-world continual post-training, providing a challenging testbed for evaluating stability–plasticity trade-offs in large-scale VLMs.

Empirical Assessment of RL Forgetting

Experiments on MRCL with current RL and SFT methods across model scales (Qwen3-VL-2B/4B/8B) reveal that:

• Both RL and SFT baselines exhibit significant catastrophic forgetting—contradicting prior claims of RL's inherent resistance.
• Classical RL approaches (e.g., GSPO, GRPO), even with KL regularization, fail to effectively resolve the stability–plasticity dilemma. In particular, GRPO’s KL constraint, when computed solely on current-task data, restricts plasticity on new tasks without robustly mitigating forgetting on prior tasks.

Empirical findings: CPO reduces forgetting by up to 13.7% (Qwen3-VL-8B) and yields up to 7.0% gains in retained pretrained model capabilities over the best competing RL methods.

Continual Policy Optimization (CPO): Replay-Free Regularization

To address catastrophic forgetting without violating privacy or incurring replay overheads, the authors propose Continual Policy Optimization (CPO), grounded in the following theoretical and methodological innovations:

Theoretical Foundation

• Behavioral KL Bound: The expected degradation in prior task performance is tightly bounded by the token-level KL divergence between the prior and updated policies on the old policy’s visitation distribution. This provides a principled target for regularization.
• Replay-Free Surrogate: Direct computation of this objective is infeasible in large-scale settings lacking access to previous task data. The authors theoretically relax the KL constraint into a parameter movement proxy, motivated by a local Fisher approximation.
• Gradient-Free Parameter Movement: The empirical movement of parameters during prior-task training serves as a noisy, but well-ordered, proxy for Fisher information, enabling efficient, gradient-free identification of parameters critical for prior-task retention.

Algorithmic Contributions

CPO employs a sparse, masked $L_1$ regularization scheme:

• After each task, CPO tracks the top $p\%$ of parameters by movement magnitude, updating a protected set.
• During subsequent training, $L_1$ movement regularization constrains the drift of these parameters, reducing behavioral deviation from previously learned policies without the need for data replay or explicit Fisher computation.
• The method is highly scalable and incurs minimal computational and memory overhead, adapting efficiently to VLMs at 2B–8B scale.

Experimental Results and Analysis

Scaling Trends:

• On Qwen3-VL-8B, CPO achieves a mean final accuracy (MFN) of 75.46%, a 13.67% increase over the best RL baseline (GSPO).
• CPO consistently maintains high mean finetune accuracy (MFT) and mean task accuracy (MTA) across all assessed model sizes.

Out-of-Domain Retention:

• Post-MRCL training, CPO demonstrably preserves (and occasionally improves) base model capabilities on challenging external benchmarks—MathVista, MMMU-Pro, POPE, CharXiv, among others.
• Contradicts previous beliefs: SFT and classical RL methods incur pronounced knowledge erosion, while CPO’s structured regularization retains and sometimes surpasses pretraining performance.

Ablation Studies:

• Substituting $L_1$ with $L_2$ regularization results in denser parameter drift and inferior retention, corroborating the theoretical expectation that sparsity is crucial for stability.

Qualitative Analysis:

• Sequential post-training trajectories show that CPO maintains stepwise multi-turn reasoning chains and visual understanding on earlier tasks, where other RL/SFT methods degrade to inconsistent or hallucinated outputs.

Theoretical and Practical Implications

This work rebuts the prevailing assumption that RL is innately resilient to forgetting and demonstrates that, under realistic continual learning settings, both RL and SFT are vulnerable without explicit regularization targeting prior-task behavior. By providing a theoretically justified, practical, and scalable surrogate objective, CPO enables robust continual adaptation for VLMs without replay.

The methodology is pertinent for privacy-sensitive and online settings where old data is unavailable, as in enterprise, healthcare, and real-time agent deployment scenarios. From a theoretical perspective, the framework unifies surrogate movement-based regularization with behavioral policy preservation, extending continual learning theory to the large-model, multimodal RL regime.

Future Prospects

Potential research directions include:

• Extending movement-based regularization primitives to other policy optimization paradigms and parameter-efficient architectures.
• Integration with dynamic task boundary detection and unsupervised continual adaptation protocols.
• Advances in the theoretical characterization of parameter movement proxies and their interaction with architectural inductive biases in LLMs and VLMs.
• Exploration of adaptive regularization coefficients and automated protected set updates, possibly leveraging meta-learning or attention over parameter importance trajectories.

Conclusion

"RL Forgets! Towards Continual Policy Optimization" provides a comprehensive critique of the intrinsic resistance of RL to catastrophic forgetting in VLM continual post-training. By establishing a challenging new benchmark (MRCL) and a theoretically motivated, scalable framework (CPO), the work substantiates that explicit replay-free regularization—rooted in parameter-movement proxies for behavioral KL preservation—substantially improves stability-plasticity trade-offs. This contribution advances the state of continual learning in large-scale, multimodal RL systems, serving as a foundation for more robust, memory-efficient deployment in dynamic environments.

Reference: "RL Forgets! Towards Continual Policy Optimization" (2607.04364)