Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning

Abstract: Continual Reinforcement Learning (CRL) for Vision-Language-Action (VLA) models is a promising direction toward self-improving embodied agents that can adapt in openended, evolving environments. However, conventional wisdom from continual learning suggests that naive Sequential Fine-Tuning (Seq. FT) leads to catastrophic forgetting, necessitating complex CRL strategies. In this work, we take a step back and conduct a systematic study of CRL for large pretrained VLAs across three models and five challenging lifelong RL benchmarks. We find that, contrary to established belief, simple Seq. FT with low-rank adaptation (LoRA) is remarkably strong: it achieves high plasticity, exhibits little to no forgetting, and retains strong zero-shot generalization, frequently outperforming more sophisticated CRL methods. Through detailed analysis, we show that this robustness arises from a synergy between the large pretrained model, parameter-efficient adaptation, and on-policy RL. Together, these components reshape the stability-plasticity trade-off, making continual adaptation both stable and scalable. Our results position Sequential Fine-Tuning as a powerful method for continual RL with VLAs and provide new insights into lifelong learning in the large model era. Code is available at github.com/UT-Austin-RobIn/continual-vla-rl.

• The paper shows that simple sequential fine-tuning with LoRA and on-policy RL matches or outperforms complex continual RL strategies across diverse benchmarks.
• The paper details a methodology using parameter-efficient adaptation and on-policy reinforcement learning to minimize catastrophic forgetting and enhance zero-shot generalization.
• The experiments reveal near-perfect task retention (≈98%) and robust adaptability, emphasizing the critical role of a large pretrained backbone in continual learning.

Vision-Language-Action Models as Natural Continual Learners: An Empirical Analysis

Introduction

This paper systematically investigates the behavior of large Vision-Language-Action (VLA) models in continual reinforcement learning (CRL) settings. Contrary to prevailing assumptions in continual learning, the work demonstrates that simple Sequential Fine-Tuning (Seq. FT) with parameter-efficient adaptation (LoRA) and on-policy RL is sufficient to attain high plasticity, minimal forgetting, and robust zero-shot generalization, often outperforming sophisticated CRL algorithms.

Figure 1: Large VLAs as natural continual learners—synergy between pretrained weights, RL, and LoRA prevents catastrophic forgetting while retaining adaptability.

Problem Formulation and Experimental Protocol

The study formalizes each task as a language-conditioned Markov Decision Process (MDP) where the policy receives an image and an instruction and outputs actions for robotic control. The CRL scenario follows the standard protocol where an agent learns a sequence of tasks $\{\mathcal{T}_1, ..., \mathcal{T}_T\}$, never revisiting previous task environments. Success is measured by average success rate (AVG), negative backward transfer (NBT) for catastrophic forgetting, forward transfer (FWT), and zero-shot generalization (ZS) to held-out tasks.

Benchmarks include LIBERO-Object, LIBERO-Spatial, LIBERO-Long, RoboCasa, and ManiSkill covering diverse manipulation domains. The study evaluates three VLA architectures and covers several parameter-efficient fine-tuning and CRL approaches, including replay-, regularization-, and parameter-isolation-based methods, as well as modern weight-merging and learning rate decoupling strategies.

Figure 2: Visualization of a representative LIBERO-object manipulation task ("pick up the chocolate pudding and place it in the basket").

Main Results: Sequential Fine-Tuning Outperforms

Seq. FT with LoRA and on-policy RL matches or surpasses the multitask oracle on most metrics and consistently outperforms regularization and replay-based approaches across all evaluated benchmarks. Notably, this configuration exhibits negligible catastrophic forgetting (NBT $\approx 0$) and strong zero-shot generalization, sometimes exceeding multitask oracle models.

Figure 3: For all tasks, Seq. FT maintains near-perfect retention—success remains at or above 98% after each new task, indicating minimal forgetting during sequential training.

Despite conventional wisdom, there is no significant need for explicit mechanisms such as expert replay, importance weighting, or parameter isolation; on the contrary, such mechanisms reduce plasticity and adaptation to new tasks. The study finds that the combination of a large pretrained backbone, LoRA adaptation, and on-policy RL fundamentally reshapes the stability-plasticity trade-off, permitting both retention and transfer.

Robustness to Environmental and Protocol Perturbations

To assess the generality of these effects, the authors conduct controlled perturbation studies:

• Camera Angle, Lighting, and Base Position: Seq. FT maintains strong performance and minimal forgetting under changes in sensor configuration and robot pose.

Figure 4: Tasks conducted under varying camera angles to test environmental robustness.

Figure 5: Tasks conducted under varying lighting conditions to test generalization and retention.

Figure 6: Tasks conducted under varied robot initial positions to assess adaptability.

• Model Variants and Benchmarks: Results generalize across VLA designs (e.g., OpenVLA, Pi-0, OpenVLA-OFT) and physical simulators (RoboCasa, ManiSkill).
• Task Orderings: Performance and retention are consistent under different orderings of task presentations.
• Extended Training: Remaining performance gaps versus multitask oracle are closed entirely by prolonging training, implying sample efficiency—not stability—is the main constraint.

Mechanistic Analysis: Why Is Seq. FT So Effective?

The study provides a thorough ablation analysis demarcating the contributions of each system component:

• On-Policy RL: Using policy gradient methods with on-policy data acts as an implicit regularization, preventing abrupt shifts in the policy and restricting the forward KL divergence from the initial policy. This stabilizes representations and reduces catastrophic forgetting even without explicit regularization.
• Large Pretrained Models: Overparameterization produces a high-dimensional null space in which most adaptation directions do not interfere with stored representations. Fisher information analysis confirms that gradient directions for new tasks have negligible projection on high-pretrained-curvature directions, explaining retention.
• LoRA Adaptation: Constraining updates to low-rank subspaces via LoRA restricts the effective degrees of freedom, thereby preventing catastrophic changes in any single layer and empirically lowering the nuclear norm and effective rank of per-layer updates.

Removing any one of these three elements (on-policy RL, large model initialization, or LoRA) restores classical catastrophic forgetting effects, confirming the necessity of their combination for stability.

Plasticity and Zero-Shot Generalization

Surprisingly, with LoRA, Seq. FT does not suffer a meaningful loss of plasticity in the RL setting, contrary to observations in supervised finetuning [biderman2024lora]. The information-theoretic explanation lies in RL's low per-episode information yield; LoRA's representational capacity is sufficient to capture needed updates. Moreover, continual adaptation even enhances zero-shot generalization relative to the multitask oracle, possibly due to implicit regularization from nonstationary training objectives.

Implications and Future Perspectives

These findings have significant implications for continual and scalable RL on large VLA models:

• Practical: There is little need for complex memory or explicit regularization mechanisms in the presence of strong pretraining and LoRA-based RL finetuning. The simple, compute- and storage-efficient Seq. FT protocol suffices for robust continual adaptation.
• Theoretical: The implicit regularization provided by on-policy RL and representational redundancy in large models mitigate the classical stability–plasticity dilemma, redefining research priorities for the next generation of embodied AI systems.
• Future Directions: Efficient adaptation strategies and improved generalization, rather than catastrophic forgetting, may emerge as the dominant challenges in embodied continual learning. Real-world transfer (e.g., sim-to-real), hierarchical and compositional generalization, and efficient exploration could become the focus of ongoing and future research.

Conclusion

This work overturns several persistent assumptions in the CRL literature by showing that large pretrained VLA models, with LoRA and on-policy RL, are natural continual learners—retaining prior knowledge, remaining plastic to new tasks, and preserving or exceeding generalization capabilities, all without elaborate architectural or training modifications. This "simple recipe" offers a new baseline for continual embodied learning in the era of foundation models.