TSN-Affinity: Similarity-Driven Parameter Reuse for Continual Offline Reinforcement Learning

Abstract: Continual offline reinforcement learning (CORL) aims to learn a sequence of tasks from datasets collected over time while preserving performance on previously learned tasks. This setting corresponds to domains where new tasks arise over time, but adapting the model in live environment interactions is expensive, risky, or impossible. However, CORL inherits the dual difficulty of offline reinforcement learning and adapting while preventing catastrophic forgetting. Replay-based continual learning approaches remain a strong baseline but incur memory overhead and suffer from a distribution mismatch between replayed samples and newly learned policies. At the same time, architectural continual learning methods have shown strong potential in supervised learning but remain underexplored in CORL. In this work, we propose TSN-Affinity, a novel CORL method based on TinySubNetworks and Decision Transformer. The method enables task-specific parameterization and controlled knowledge sharing through a RL-aware reuse strategy that routes tasks according to action compatibility and latent similarity. We evaluate the approach on benchmarks based on Atari games and simulations of manipulation tasks with the Franka Emika Panda robotic arm, covering both discrete and continuous control. Results show strong retention from sparse SubNetworks, with routing further improving multi-task performance. Our findings suggest that similarity-guided architectural reuse is a strong and viable alternative to replay-based strategies in a CORL setting. Our code is available at: https://github.com/anonymized-for-submission123/tsn-affinity.

• The paper introduces TSN-Affinity, a framework that leverages sparse TinySubNetworks and affinity metrics for effective parameter reuse in continual offline RL.
• The paper implements dynamic routing strategies based on action and latent similarity, achieving near-zero forgetting in multi-task Atari benchmarks.
• The paper demonstrates that architectural reuse surpasses traditional replay-based methods, offering scalable solutions to fixed-dataset reinforcement learning challenges.

TSN-Affinity: Similarity-Driven Architectural Reuse in Continual Offline RL

Problem Formulation and Motivation

The paper "TSN-Affinity: Similarity-Driven Parameter Reuse for Continual Offline Reinforcement Learning" (2604.25898) addresses the continual offline RL (CORL) paradigm, wherein agents learn a sequence of tasks solely from fixed datasets without any on-policy environment interaction. This scenario is increasingly pertinent in domains where live interactions are expensive, unsafe, or infeasible. The key challenge is twofold: optimizing for new tasks while ensuring retention on previously acquired skills, under constraints of data access, capacity, and stability.

Replay-based continual learning approaches are the dominant baseline but suffer from memory overhead and distribution mismatches between replayed samples and current task policies. Architectural approaches, prominent in continual supervised learning, have shown strong forgetting resistance but remain underexplored in CORL. The paper proposes TSN-Affinity—a framework that leverages sparse TinySubNetworks (TSN) parameterization, integrated with a Decision Transformer backbone, and introduces affinity-guided parameter reuse and routing.

Figure 1: Overview of the proposed TSN-Affinity method, featuring affinity-driven copy selection and task-specific mask allocation.

Methodology

TSN-Affinity operates by constructing sparse, task-specific subnetworks for each incoming task. For each task, the method evaluates whether to reuse previously learned model copies or to allocate new ones, based on similarity scores informed by task-specific affinity metrics. Specifically, affinity scores are derived via action-level compatibility (policy output similarity), latent representation similarity (encoder statistics), or hybrid combinations.

The routing decision centers on minimizing interference and maximizing transfer. If the affinity between the new and prior tasks is above a threshold, parameter reuse is triggered: parameters are shared (and remain frozen), while only task-specific mask scores and new parameters are trained. Otherwise, a new copy is spawned. Weight protection is enforced rigorously, ensuring strict parameter immutability when reused.

Three routing variants are investigated:

• Action Affinity: Cross-entropy (discrete) or masked MSE (continuous) between prior policy outputs and new task data.
• Latent Affinity: Symmetric KL divergence between task-conditional encoder statistics.
• Hybrid Affinity: Weighted combination of normalized action and latent scores.

Reduced variants include TSN-ReplayKL (routing via replay-memory similarity) and TSN-Core (no routing, single model copy).

Experimental Design

Two benchmark suites are utilized:

1. Atari Continual Decision Making: Five discrete-action ALE games (Breakout, Alien, Atlantis, Boxing, Centipede), with high-dimensional visual observations.
2. Panda Continual Manipulation: Three continuous-control PandaGym robotic manipulation tasks (Reach, Push, Pick-and-Place), featuring heterogeneous action spaces and increasing task complexity.

The evaluation protocol reports both aggregate multi-task performance and detailed metrics on preservation (forgetting scores) and transfer (performance gap-to-target).

Figure 2: Additional Atari learning curves, showing affinity-based TSN variants achieving near-zero-forgetting across tasks.

Figure 3: Additional Panda learning curves for qualitative comparison; main metrics discussed are gap-to-target return and retention.

Results and Analysis

Atari Benchmark

Sparse architectural methods demonstrate absolute retention: TSN-based variants eliminate catastrophic forgetting completely. TSN-Core and TSN-ReplayKL outperform replay baselines, with affinity-based routing (action and latent) achieving the highest normalized mean returns (up to 89.7%), exceeding single-task Decision Transformer reference performance.

Dense baselines (naive, EWC, SI) exhibit severe forgetting, even with cumulative replay, which, despite improvement, does not match sparse TSN performance. Routing threshold analysis reveals that conservative reuse (more copies, stricter affinity) improves aggregate performance, although at increased capacity cost.

Panda Benchmark

Transfer and retention are substantially harder. Replay-memory KL routing is unreliable in this setting due to heterogeneous state/action spaces. TSN-Core outperforms dense baselines but does not match cumulative replay. Affinity-driven routing (TSN-Affinity-L) achieves best gap-to-target scores and zero forgetting—yet only with several model copies, indicating a performance-capacity tradeoff.

Task complexity and intrinsic offline RL difficulty are evident: even single-task baselines do not attain optimal returns for Push and Pick-and-Place. Thus, architectural reuse addresses interference but does not bypass data/model mismatch inherent to offline RL.

Theoretical and Practical Implications

The results reinforce the efficacy of architectural continual learning in offline RL, especially when replay is suboptimal due to memory or distribution mismatch. Task-specific parameterization and affinity-driven reuse enable strict retention and controlled transfer. The affinity routing protocol introduces an RL-aware mechanism for dynamic architectural adaptation, offering improved transfer when tasks are sufficiently similar.

Strict weight protection and copy-level allocation are critical in maintaining backward compatibility, a challenge emphasized in CORL. Importantly, affinity scores rooted in control-relevant metrics outperform naive observational similarity.

Future Directions in Continual RL

Potential extensions include:

• Automated Routing Thresholds: Dynamic selection of acceptance criteria for reuse decisions, reducing dependence on manual sweeps.
• Generalization to Complex Formats: Scaling affinity routing to larger and more diverse continual RL benchmarks, including multi-modal input/action spaces.
• Online Continual Learning: Adapting affinity routing for settings with on-policy or simulated environment interaction.
• Hybrid Architectural Strategies: Incorporating adaptive pruning, quantization, and modularity to balance performance, efficiency, and capacity.

Conclusion

TSN-Affinity establishes an architectural alternative to replay-based continual learning in offline RL. Through sparse task-specific subnetworks and similarity-guided parameter reuse, the method mitigates forgetting and facilitates transfer in sequential learning scenarios. Affinity-based routing emerges as a principled criterion for architectural adaptation, outperforming replay-memory heuristics in continuous-control domains. The framework poses new questions regarding scalability, automated routing, and hybrid continual learning, with broad implications across adaptive RL systems.