1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities

Abstract: Scaling up self-supervised learning has driven breakthroughs in language and vision, yet comparable progress has remained elusive in reinforcement learning (RL). In this paper, we study building blocks for self-supervised RL that unlock substantial improvements in scalability, with network depth serving as a critical factor. Whereas most RL papers in recent years have relied on shallow architectures (around 2 - 5 layers), we demonstrate that increasing the depth up to 1024 layers can significantly boost performance. Our experiments are conducted in an unsupervised goal-conditioned setting, where no demonstrations or rewards are provided, so an agent must explore (from scratch) and learn how to maximize the likelihood of reaching commanded goals. Evaluated on simulated locomotion and manipulation tasks, our approach increases performance on the self-supervised contrastive RL algorithm by $2\times$ - $50\times$, outperforming other goal-conditioned baselines. Increasing the model depth not only increases success rates but also qualitatively changes the behaviors learned. The project webpage and code can be found here: https://wang-kevin3290.github.io/scaling-crl/.

• The paper demonstrates that increasing network depth up to 1024 layers results in dramatic performance improvements, with up to 50x gains in humanoid tasks.
• The paper leverages residual connections, layer normalization, and Swish activation to ensure stable training and effective gradient propagation in ultra-deep networks.
• The study reveals that deeper networks enable emergent qualitative behaviors, such as acrobatic maneuvers, which improve policy learning in complex, unsupervised environments.

Scaling Depth in Self-Supervised RL: A Technical Insight

Introduction

The paper "1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities" (2503.14858) explores the frontier of reinforcement learning (RL) by challenging the common reliance on shallow neural networks and proposing the substantial scaling of network depth. While breakthroughs in language and vision have been achieved through the expansion of self-supervised learning models, scaling up in RL remains relatively unexplored. This study investigates the potential of self-supervised RL with deep networks extending up to 1024 layers to enhance goal-reaching performance in unsupervised settings across various tasks.

Methodology and Experimental Design

Network Architecture

The authors employ a combination of architectural innovations to stabilize training procedures of deeply scaled networks. Specifically, the networks incorporate residual connections, layer normalization, and Swish activation functions. These improvements are borrowed from prior deep learning advancements to ensure effective gradient propagation and training stability in networks.

Experimental Setup

Experiments were conducted in an unsupervised goal-conditioned setting, where agents were tasked with reaching specified goals based on their own exploration, absent of any external demonstrations or rewards. These experiments were situated across simulated environments designed to test locomotion and manipulation skills in challenging configurations which include the Ant and Humanoid mazes.

Figure 1: Scaling network depth yields performance gains across a suite of locomotion and manipulation tasks, with marked improvements at critical depths specific to each environment.

Results and Analysis

Performance Enhancement via Depth Scaling

The results reveal a stark enhancement in performance upon significant scaling of network depth. For example, in humanoid-based tasks, depth scaling introduces performance improvements from twofold to fiftyfold. The scaling does not progress linearly; rather, performance leaps are often realized at critical network depths, indicating an emergent capability when certain architectural complexities are reached.

Figure 2: Incrementally increasing depth results in marginal performance gains (left). However, once a critical threshold is reached, performance improves dramatically (right).

Neural Expressivity and Batch Size

Deep networks demonstrated superior expressivity, allowing for qualitatively distinct policies to emerge. The exploration capabilities and state-space coverage are augmented, suggesting that increased network depth enables more nuanced policy learning. Furthermore, deeper networks were able to benefit from larger batch sizes, a scaling behavior traditionally advantageous in large model regimes of other domains.

Figure 3: Deeper networks unlock batch size scaling, providing further performance improvements as depth increases from 4 to 64.

Emergent Properties

Distinct qualitative behaviors were visible as a function of network depth. In the Humanoid maze, for instance, deep networks began to exhibit unique strategies like acrobatic maneuvers to overcome obstacles, an indicative mark of emergent learning properties uncharacteristic in shallower networks.

Theoretical and Practical Implications

The capability for deep networks to effectively map state space with increased detail means RL could see a paradigm shift similar to vision and NLP, where model scaling is a norm. Practically, the findings could propel advancements in robotics requiring fine-tuned manipulation and responsive learning. Theoretically, these results inspire a potential reconsideration of RL architecture guidelines, advocating for testing of network depth as a performance lever.

Future Directions

Future exploration could focus on integrating these findings within distributed RL to leverage computational infrastructures fully, and examining the synergistic effect of scaling both depth and width in tandem. An additional avenue could involve investigating supplementary self-supervised methods where depth scaling might reveal different or complementary competencies.

Conclusion

The paper effectively posits that substantial scaling of network depth revitalizes the capacity for self-supervised RL to achieve unprecedented levels of performance. This operates not only as an engineering exploit to amplify existing architectures but also as a profound commentary on the unexplored depth of capacity and representation in RL networks. As computational resources continue to rise, so too may the plausibility and prevalence of ultra-deep networks in reinforcement learning applications.