Shared LoRA Subspaces for almost Strict Continual Learning

Abstract: Adapting large pretrained models to new tasks efficiently and continually is crucial for real-world deployment but remains challenging due to catastrophic forgetting and the high cost of retraining. While parameter-efficient tuning methods like low rank adaptation (LoRA) reduce computational demands, they lack mechanisms for strict continual learning and knowledge integration, without relying on data replay, or multiple adapters. We propose Share, a novel approach to parameter efficient continual finetuning that learns and dynamically updates a single, shared low-rank subspace, enabling seamless adaptation across multiple tasks and modalities. Share constructs a foundational subspace that extracts core knowledge from past tasks and incrementally integrates new information by identifying essential subspace directions. Knowledge from each new task is incorporated into this evolving subspace, facilitating forward knowledge transfer, while minimizing catastrophic interference. This approach achieves up to 100x parameter reduction and 281x memory savings over traditional LoRA methods, maintaining performance comparable to jointly trained models. A single Share model can replace hundreds of task-specific LoRA adapters, supporting scalable, asynchronous continual learning. Experiments across image classification, natural language understanding, 3D pose estimation, and text-to-image generation validate its effectiveness, making Share a practical and scalable solution for lifelong learning in large-scale AI systems.

• The paper introduces Share, a method that leverages shared LoRA subspaces to enable replay-free continual learning with minimal parameter overhead.
• It utilizes SVD-based initialization and closed-form projection to update task-specific coefficients while preserving past knowledge.
• Empirical results demonstrate that Share achieves up to 100x parameter efficiency and robust performance across language, vision, and generative tasks.

Shared LoRA Subspaces for Almost Strict Continual Learning

Introduction and Context

The paper "Shared LoRA Subspaces for almost Strict Continual Learning" (2602.06043) addresses the problem of continual adaptation of large pretrained models (e.g., LLMs, VLMs, diffusion models) with minimal parameter overhead while maintaining knowledge from previous tasks. The work explicitly targets the limitations of parameter-efficient tuning protocols such as LoRA, which, despite their computational benefits, do not innately provide mechanisms for strict continual learning (CL) or scalable adapter integration. The presented method, Share, proposes a novel shared subspace adaptation framework designed to enable replay-free, nearly strict CL with extreme efficiency, cross-domain applicability, and robust knowledge transfer.

Theoretical Motivation and Methodological Framework

Foundational Subspace Hypothesis

The central observation of the paper is that LoRA adapters trained across correlated tasks converge toward a low-dimensional shared subspace in weight space. Empirical evidence (via linear CKA) reveals that models initialized with different task permutations rapidly converge to such a foundational subspace during sequential CL, achieving >0.95 similarity after a handful of tasks.

Figure 1: CKA analysis demonstrates monotonic convergence of LoRA subspaces to a shared foundation during continual adaptation.

This supports the universal weight subspace hypothesis, and motivates the design of Share, which aims to construct and continually evolve a single shared subspace capturing essential directions for downstream adaptation, thus minimizing catastrophic forgetting and maximizing resource efficiency.

Share: Continual Subspace Adaptation

Share maintains a set of principal basis vectors (frozen during most adaptation) and learns task-specific coefficients for new tasks or incoming adapters. The method comprises:

1. Initialization: From available LoRA adapters or data, principal factors are extracted using SVD to form the foundational subspace.
2. Continual Adaptation: For new tasks or adapters, only the coefficients associated with a small subset of subspace directions (or newly added temporary basis) are updated.
3. Merging/Update: Newly learned directions and coefficients are merged (via analytical SVD updates), and all prior knowledge is reprojected onto the updated subspace; coefficients are recalculated via closed-form projection using the Moore-Penrose pseudo-inverse.

This approach ensures strictly bounded parameter and memory complexity as task count grows, with empirical ablation showing that pseudo-rank $p=1$ and moderate $k$ achieve optimality.

Figure 2: Schematic presentation of the continual adaptation and merging steps in Share, highlighting the minimal adaptation space.

Theoretical Analysis

The paper formalizes the error bounds for the incremental construction of the shared subspace and the resulting approximation error for newly arriving tasks. The incremental SVD approach guarantees that the loss in cross-task knowledge is dominated by the eigen-spectrum truncation beyond the top-$k$ directions, with expected approximation error growing sub-linearly with task count under reasonable similarity assumptions. The generalization gap for new tasks is tightly controlled relative to unconstrained low-rank adaptation as long as new task solutions reside within the constructed subspace, otherwise expanding flexibly.

Empirical Evaluation

Cross-Modal Applicability

Share is evaluated across language (GLUE benchmark), vision (CIFAR-100, Food-101, Caltech-101, Flowers-102), structured geometry (3D pose estimation on Pascal3D+), and generative domains (text-to-image with Flux, large-scale LoRA merging in LLMs).

Continual Task Sequences and Subspace Tracking

Share equipped with $O(10^4)$ parameters matches or exceeds fully fine-tuned or non-CL LoRA models requiring $O(10^6-10^7)$ parameters. Parameter efficiency approaches 100x in typical settings, and empirical memory savings exceed 250x, all while maintaining high average accuracy and dramatically reducing forgetting compared to rehearsal-based and prompt-based methods.

Notably, the reconstructed error and CKA alignment of the foundational subspace both stabilize and converge through sequential learning, showing the robustness and dynamic refinement of the principal directions.

Figure 3: Share achieves low reconstruction error over all tasks' LoRA adapters, confirming subspace sharing.

Figure 4: CKA similarity of Share's evolving factor space indicates effective incorporation of new information over time.

Large-Scale Adapter Merging

Share supports continual, asynchronous merging of hundreds of LoRA adapters (e.g., Mistral-7B scenario), compressing them into a single set of principal factors. The method maintains 90–99% of adapter performance in both in-distribution and OOD settings, vastly outperforming task-specific or batch-merging competitors such as TIES in zero-shot or cross-domain scenarios.

Figure 5: Share integrates hundreds of LoRA adapters continually, maintaining low reconstruction error and high downstream performance.

Generative and Multimodal Tasks

For diffusion/text-to-image models, Share achieves competitive sample quality against task-specific LoRA finetuning with a single set of factors—even when handling multiple stylistically diverse tasks—demonstrating broad applicability and generalization capacity for multimodal adaptation.

Figure 6: Comparison of text-to-image results: Share (continual, shared subspace) vs independently trained LoRA adapters.

Practical and Theoretical Implications

Share redefines the lower-bound of parameter and memory requirements for continual adaptation in large foundation models, providing a scalable, robust tool for research and deployment in continuously evolving environments. Its ability to perform replay-free, domain-agnostic, and highly compressive CL enables rapid AI personalization, multi-user deployment, federated or decentralized learning, and environmentally sustainable training regimes.

Theoretically, Share substantiates the centrality of shared weight structures and supports the development of new architectures and learning protocols that leverage universal subspaces for efficient multi-task and lifelong learning in deep networks.

Future Directions

Outstanding directions include extending Share to enable knowledge integration across heterogeneous architectures, handling cross-modal transfer learning, relaxing strict backbone homogeneity, and integrating direct metric learning for more expressive subspace control. Further work may focus on improving OOD generalization in absence of task-aligned adapters, as well as low-resource initialization, and exploring federated settings with privacy guarantees.

Conclusion

Share establishes a systematic and theoretically justified protocol for parameter-efficient continual finetuning by leveraging the emergent shared subspace across LoRA adapters. The method achieves nearly strict continual learning—avoiding catastrophic forgetting, data replay, and adapter proliferation—while matching or surpassing strong task-specific adaptation baselines. The presented empirical evidence and analytical framework mark Share as a pivotal advance for robust, scalable, and sustainable adaptation of modern foundation models (2602.06043).