Multi-task Deep Reinforcement Learning with PopArt

Abstract: The reinforcement learning community has made great strides in designing algorithms capable of exceeding human performance on specific tasks. These algorithms are mostly trained one task at the time, each new task requiring to train a brand new agent instance. This means the learning algorithm is general, but each solution is not; each agent can only solve the one task it was trained on. In this work, we study the problem of learning to master not one but multiple sequential-decision tasks at once. A general issue in multi-task learning is that a balance must be found between the needs of multiple tasks competing for the limited resources of a single learning system. Many learning algorithms can get distracted by certain tasks in the set of tasks to solve. Such tasks appear more salient to the learning process, for instance because of the density or magnitude of the in-task rewards. This causes the algorithm to focus on those salient tasks at the expense of generality. We propose to automatically adapt the contribution of each task to the agent's updates, so that all tasks have a similar impact on the learning dynamics. This resulted in state of the art performance on learning to play all games in a set of 57 diverse Atari games. Excitingly, our method learned a single trained policy - with a single set of weights - that exceeds median human performance. To our knowledge, this was the first time a single agent surpassed human-level performance on this multi-task domain. The same approach also demonstrated state of the art performance on a set of 30 tasks in the 3D reinforcement learning platform DeepMind Lab.

• The paper introduces PopArt normalization in an actor-critic framework to balance learning across diverse tasks.
• The method leverages the IMPALA architecture, yielding a 110% human-normalized score on Atari-57 and 72.8% on DeepMind Lab.
• The study shows that task-specific normalization enhances generalization and efficiency in multi-task deep reinforcement learning.

Multi-task Deep Reinforcement Learning with PopArt

The paper "Multi-task Deep Reinforcement Learning with PopArt" addresses the significant challenge of training reinforcement learning (RL) agents that can proficiently handle multiple tasks simultaneously, contrasting with the traditional approach of single-task training. This study is particularly important as it seeks to enhance the flexibility and applicability of RL agents in solving diverse problems within a unified framework.

Problem Context and Objectives

The primary objective is to develop a reinforcement learning algorithm that can efficiently learn multiple sequential-decision tasks while achieving robust performance across all of them. Multi-task learning in RL is complicated by the fact that tasks can have vastly different characteristics, such as reward magnitudes and densities, which may impede uniform learning progress. A key challenge is managing how different tasks influence the learning updates, preventing more salient tasks from overshadowing others.

Methodology

To tackle these challenges, the authors propose the use of PopArt (a normalisation technique) within an actor-critic RL framework. This method dynamically normalises the contributions of each task to the learning updates, enabling a balanced training process where each task has a similar impact on the agent's learning trajectory. The actor-critic architecture, robust in handling value-based predictions and policy updates, is well-suited for such a scale-invariant normalisation approach.

The agent architecture employed is the Importance Weighted Actor-Learner Architecture (IMPALA), which supports efficient parallel learning by dividing the task across multiple actors. Notably, the multi-task RL agent is trained on a wide range of tasks, i.e., 57 Atari games and 30 tasks in DeepMind Lab, using a single policy and value function with task-specific normalisation capabilities.

Results

The numerical results from the paper are compelling. The PopArt-empowered IMPALA agent achieves a median human-normalized score of 110% on Atari-57 and 72.8% on DmLab-30, significantly outperforming the baseline IMPALA, which scored 59.7% and 60.6%, respectively. The inclusion of a single set of weights for all tasks, yet achieving human-level or better performance across them, highlights the method's capacity for generalisation beyond single-task agents.

Implications

From a theoretical standpoint, this research demonstrates that task-specific normalisation strategies, like PopArt, are vital in promoting fair and effective learning in a multi-task context. Practically, this advancement could enable more versatile applications of RL where training resources are constrained or when it is desirable to have a single agent capable of tackling a diversity of tasks without task-specific modifications.

Future Directions

The paper opens several avenues for future exploration. Incorporating PopArt normalisation with other multi-task strategies, such as active task sampling and policy distillation, could further enhance training efficiency and policy generality. Additionally, scaling this approach to even more complex task sets with higher-dimensional action spaces remains a critical frontier.

In essence, the approach proposed in the paper serves as a significant step forward in multi-task deep reinforcement learning, providing a robust framework that facilitates simultaneous learning over extensive and disparate tasks using a unified model. This achievement not only sets a benchmark for future multi-task RL research but also expands the potential applicability of RL agents in real-world scenarios where varied task execution is essential.