Statistical Context Detection for Deep Lifelong Reinforcement Learning

Abstract: Context detection involves labeling segments of an online stream of data as belonging to different tasks. Task labels are used in lifelong learning algorithms to perform consolidation or other procedures that prevent catastrophic forgetting. Inferring task labels from online experiences remains a challenging problem. Most approaches assume finite and low-dimension observation spaces or a preliminary training phase during which task labels are learned. Moreover, changes in the transition or reward functions can be detected only in combination with a policy, and therefore are more difficult to detect than changes in the input distribution. This paper presents an approach to learning both policies and labels in an online deep reinforcement learning setting. The key idea is to use distance metrics, obtained via optimal transport methods, i.e., Wasserstein distance, on suitable latent action-reward spaces to measure distances between sets of data points from past and current streams. Such distances can then be used for statistical tests based on an adapted Kolmogorov-Smirnov calculation to assign labels to sequences of experiences. A rollback procedure is introduced to learn multiple policies by ensuring that only the appropriate data is used to train the corresponding policy. The combination of task detection and policy deployment allows for the optimization of lifelong reinforcement learning agents without an oracle that provides task labels. The approach is tested using two benchmarks and the results show promising performance when compared with related context detection algorithms. The results suggest that optimal transport statistical methods provide an explainable and justifiable procedure for online context detection and reward optimization in lifelong reinforcement learning.

• The paper presents SWOKS, a novel algorithm that uses sliced Wasserstein distances and Kolmogorov-Smirnov tests to accurately detect task changes in online deep reinforcement learning.
• It employs a rollback mechanism to maintain task-specific policies, effectively mitigating catastrophic forgetting by reverting to previous checkpoints upon detecting task shifts.
• Experimental validations in CT-graph, Minigrid, and Half-Cheetah environments demonstrate that SWOKS outperforms existing methods by ensuring robust policy optimization in complex lifelong RL scenarios.

An Analytical Overview of SWOKS for Context Detection in Lifelong Learning

The paper presents a novel algorithm, Sliced Wasserstein Online Kolmogorov-Smirnov (SWOKS), for task detection and policy optimization in online deep reinforcement learning (RL) settings. Lifelong reinforcement learning (LRL) involves training agents to handle multiple sequential tasks, mitigating the well-known issue of catastrophic forgetting. Standard approaches struggle with inferring task labels from online experiences, especially given changes in transition or reward functions, which SWOKS addresses effectively.

Methodology

The SWOKS algorithm detects task changes by leveraging statistical methods on the latent action-reward spaces derived from the data streams. The essential technique involves computing the Wasserstein distance (WD), approximated using the Sliced Wasserstein Distance (SWD), between sets of experiences to analyze shifts in data distributions. These distances provide inputs for the Kolmogorov-Smirnov (KS) statistical test, which validates whether the current data belongs to a known task or indicates a new or previously observed task. The algorithm controls for false positives with a tuned multiplicative parameter $\beta$ to adjust the reference SWD.

To maintain multiple task-specific policies, SWOKS incorporates a rollback mechanism where the current policy states are periodically saved and reverted based on the detected task changes. This ensures isolated learning for each task, enabling the deployment of the correct policy by reverting to appropriate checkpoints.

Experimental Validation

SWOKS is benchmarked against established algorithms across various environments. In the CT-graph environment, results indicate that SWOKS sustains consistent performance across multiple tasks by effectively using the modulating masks for policy differentiation, whereas TFCL (Task-Free Continual Learning) struggles due to task interference. In the Minigrid environment, SWOKS demonstrates robust task detection despite partial learning failures, indicating its efficacy in environments with varying observation spaces.

In the continuous action space of the Half-Cheetah Mujoco environment, SWOKS outperforms the Model-Based Context Detection (MBCD) and Replay-based Recurrent Reinforcement Learning (3RL) algorithms. SWOKS's structured policy separation and rollback mechanism prevent the negative impacts of task interference, and its KS statistical test yields discerning task identification.

Implications and Future Directions

The combination of optimal transport distances with non-parametric statistical hypothesis testing provides an explainable and statistically grounded foundation for task detection in LRL scenarios. This approach makes SWOKS adaptable to diverse environments by tuning the statistical significance threshold ($\alpha$) and correction factor ($\beta$). The method's scalability to various domains, including those with complex reward structures and continuous action spaces, opens promising avenues for fine-tuning RL systems for real-world applications.

However, the need for parameter tuning and sequential policy examination for task re-detection in large task sets poses challenges. Future research could explore adaptive $\beta$ parameters based on the standard deviation of the data, enhancing the algorithm's robustness across domains. Additionally, integrating clustering techniques with the SWD framework for narrowing down potential policies could further optimize the re-detection mechanism, reducing the computational load.

Conclusion

The SWOKS algorithm represents a significant advancement in task detection and policy optimization for lifelong learning in reinforcement learning. Its methodical approach combining SWD and KS tests ensures precise task changes detection, allowing efficient lifelong learning without catastrophic forgetting. The empirical results validate its competency across multiple benchmarks, setting the stage for further developments to handle more intricate and large-scale RL scenarios.