Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning

Abstract: A hallmark of intelligent agents is the ability to learn reusable skills purely from unsupervised interaction with the environment. However, existing unsupervised skill discovery methods often learn entangled skills where one skill variable simultaneously influences many entities in the environment, making downstream skill chaining extremely challenging. We propose Disentangled Unsupervised Skill Discovery (DUSDi), a method for learning disentangled skills that can be efficiently reused to solve downstream tasks. DUSDi decomposes skills into disentangled components, where each skill component only affects one factor of the state space. Importantly, these skill components can be concurrently composed to generate low-level actions, and efficiently chained to tackle downstream tasks through hierarchical Reinforcement Learning. DUSDi defines a novel mutual-information-based objective to enforce disentanglement between the influences of different skill components, and utilizes value factorization to optimize this objective efficiently. Evaluated in a set of challenging environments, DUSDi successfully learns disentangled skills, and significantly outperforms previous skill discovery methods when it comes to applying the learned skills to solve downstream tasks. Code and skills visualization at jiahenghu.github.io/DUSDi-site/.

• The paper presents DUSDi, which decomposes skills into distinct components targeting individual state factors for efficient hierarchical RL.
• It employs a mutual-information-based objective and value factorization to minimize interference and optimize skill specificity.
• Experimental results demonstrate enhanced learning efficiency and superior performance across environments from 2D navigation to 3D robotics.

Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning

The paper, "Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning," presents a novel method aimed at addressing the challenges faced in unsupervised skill discovery through disentanglement. Traditional unsupervised skill discovery methods often produce entangled skills, which negatively affects the potential for efficient skill reuse in hierarchical reinforcement learning (HRL). The paper introduces Disentangled Unsupervised Skill Discovery (DUSDi), a method that disentangles skill components to affect singular state factors, enhancing their applicability in downstream tasks.

Key Contributions

1. Disentangled Skill Components: DUSDi emphasizes the decomposition of skills into separate components, each addressing only one dimension of the state space. This approach not only allows for concurrent composition of skill components but also facilitates efficient chaining in hierarchical RL settings.
2. Mutual Information Objective: The method centers around a novel mutual-information-based objective to enforce disentangled skills, optimizing the diversity and specificity of skill influence on individual state factors. By directly targeting individual state factors, DUSDi ensures minimal interference across unrelated dimensions.
3. Value Factorization: To efficiently handle the disentanglement process, DUSDi introduces value factorization, optimizing the proposed objective in a scalable manner. This technique enhances skill learning by reducing variance associated with aggregated evaluations of multiple skill components.

Experimental Results

Empirical evaluations demonstrate DUSDi's superior performance in diverse environments. It is shown to outperform its precursors significantly when applied to complex downstream tasks. The environments include a 2D agent navigation space, the DMC walker domain, a large-scale multi-agent setting, and a 3D simulated robotics domain. These evaluations underline DUSDi's ability to learn and deploy disentangled skills effectively across various scenarios.

Practical Implications

The implications for hierarchical reinforcement learning are significant. By providing a structured skill space, DUSDi enhances exploration efficiency in task environments, leading to superior learning performance with reduced sample inefficiency. The technique offers practical benefits in robotics and other domains that demand concurrent skill execution and chaining.

Theoretical Implications and Future Directions

The disentangled framework proposed in DUSDi highlights potential pathways for future research in unsupervised RL, particularly focusing on enhancing learning efficiency and restructuring latent spaces for skill discovery. The proposal of restricting skill influence to distinct state factors through disentanglement may lead to advances in domain-specific skill discovery and task decomposition.

Conclusion

DUSDi represents a considerable step forward in unsupervised skill discovery by leveraging state factorization to drive efficiency in hierarchical RL. The method's integration of mutual-information-based objectives with value factorization creates a robust framework capable of learning versatile and highly applicable skills. As AI continues to navigate increasingly complex task environments, innovations like DUSDi will be crucial in pushing the boundaries of what unsupervised skill discovery can achieve.