Unsupervised Skill Discovery in RL

• Unsupervised Skill Discovery is a reinforcement learning methodology that autonomously learns diverse, latent-conditioned behaviors without relying on external reward signals.
• It employs methods like mutual information maximization, contrastive learning, and distance-maximizing objectives to encourage skill diversity and disentanglement.
• Structured USD techniques enable hierarchical control and rapid adaptation in tasks ranging from robotics to simulation through skill factorization and intrinsic rewards.

Unsupervised Skill Discovery (USD) refers to a family of reinforcement learning (RL) methodologies aimed at autonomously learning a broad set of diverse, reusable behaviors—referred to as “skills”—without recourse to externally provided, task-specific reward functions. The goal is to generate latent-conditioned policies $\pi(a \mid s, z)$ that can serve as primitives for rapid adaptation and efficient hierarchical control in downstream tasks. USD spans several theoretical and algorithmic designs, evolving from mutual-information–based approaches to distance-maximizing, contrastive, factorized, and symmetry-aware frameworks, each with distinct mechanisms for encouraging diversity, disentanglement, and utility of the discovered skills.

1. Objectives and Theoretical Foundations

USD formalizes skill learning as optimizing policies conditioned on a latent variable $z \in \mathcal{Z}$, drawn from a fixed prior $p(z)$, such that each skill induces distinct, predictable behavior in the absence of external task rewards. The core unsupervised objectives fall into three main categories:

• Mutual Information Maximization: Early approaches, such as DIAYN, maximize $I(S; Z) = H(Z) - H(Z|S)$, encouraging each skill to produce states that are easily identifiable by a skill discriminator $q_\phi(z|s)$ (Imagawa et al., 2023).
• Distance-Maximizing (Wasserstein Dependency Measures): These methods replace the KL-based MI objective by maximizing the expected alignment between skill and state-embedding displacements, often imposing Lipschitz continuity for dynamic and far-reaching behaviors (Park et al., 2022).
• Contrastive and Ensemble Objectives: Approaches utilizing contrastive learning or ensemble value functions define skill diversity and coverage via InfoNCE-type losses or particle-based entropy estimators, sometimes bypassing direct MI estimation (Laskin et al., 2022, Yang et al., 2023, Bai et al., 2024).

Several objectives impose additional structural or quality constraints, including state-space factorization, symmetry invariance, controllability-aware distances, and offline imitation constraints (Hu et al., 2024, Chang et al., 20 Jan 2026, Park et al., 2023, Vlastelica et al., 2023).

2. Algorithmic Methodologies

USD frameworks instantiate the above objectives with differing architecture and optimization strategies:

• Skill-Conditioned Policy Representation: Policies $\pi(a|s, z)$ are implemented as either shared networks conditioned on $z$ (continuous or discrete, often concatenated to $s$), or independent policies per skill (incremental or ensemble methods) (Shafiullah et al., 2022, Bai et al., 2024).
• Skill Embedding and Discriminator Architectures: Depending on the method, discriminators $q_\phi(z|s)$ range from simple classifiers to von Mises–Fisher distributions (e.g., for $z$ on the unit sphere), or contrastive encoders with InfoNCE/SimCLR-type objectives (Imagawa et al., 2023, Yang et al., 2023).
• Intrinsic Reward Construction: The instantaneous intrinsic reward may be defined as a log-probability output of a discriminator, the inner product between skill and state-embedding increments, or particle-based entropy/density metrics depending on exploration strategy (Park et al., 2022, Laskin et al., 2022).
• Optimization Procedures: Policy learning is most often performed with entropy-regularized actor–critic algorithms (SAC, DDPG), combined with alternately updated discriminators/encoders. Some approaches additionally optimize a Lagrangian multiplier to enforce Lipschitz or state-space constraints (Atanassov et al., 2024).
• Augmentations and Meta-Structures: Innovations include symmetry-based data augmentation and equivariant embeddings (Chang et al., 20 Jan 2026, Cathomen et al., 27 Aug 2025), skill-tree expansion (for incremental coverage) (Kamienny et al., 2021), and regret-aware min–max games between a skill generator and agent policy (Zhang et al., 26 Jun 2025).

3. Factorization, Disentanglement, and Structured Representations

Recent work recognizes that holistic reward or MI objectives lead to entangled skills, which impede hierarchical or compositional control. Structured USD methods address this via:

• State and Skill Space Factorization: Partitioning the state as $S = S^1 \times \cdots \times S^N$ and skills as $Z = Z^1 \times \cdots \times Z^N$ enables each skill variable to control a separate environment factor. Methods such as DUSDi and SUSD maximize per-factor MI/disentanglement, promote independent skill components, and support efficient value factorization and downstream HRL chaining (Hu et al., 2024, Hosseini et al., 2 Feb 2026, Cathomen et al., 27 Aug 2025).
• Dynamic Focus via Curiosity or Underexploration Weights: Curiosity-weighted rewards and density models monitor which state factors remain poorly covered, adaptively steering exploration towards those underexplored entities (Hosseini et al., 2 Feb 2026).
• Ensemble and Partition-Based Exploration: Ensemble critics, prototype-based clustering, and per-skill entropy maximization in partitioned state space further encourage local exploration and mitigate skill collapse (Bai et al., 2024).

4. Constraints, Symmetry, and Real-World Deployability

USD frameworks often impose additional structural constraints:

• Distance and Norm Constraints: Many advances enforce that latent distances in the skill space correspond to interpretable, controllable state differences, using Lipschitz bounds, norm-matching objectives, or controllability-aware learned metrics. This makes discovered skills dynamic and task-relevant (Atanassov et al., 2024, Park et al., 2023).
• Symmetry and Group Invariance: By explicitly embedding group symmetries (rotational, reflection, etc.) into both policy and reward structure, approaches such as GISD and Divide, Discover, Deploy eliminate redundant behaviors, improve coverage efficiency, and yield interpretable, morphology-aware skills (Chang et al., 20 Jan 2026, Cathomen et al., 27 Aug 2025).
• Safety and Style Priors: Practical robotics applications incorporate style factors and regularization rewards for safe, robust behaviors. Penalizing unsafe transitions or promoting desirable trajectories via instruction learning, style factors, or cost critics allows zero-shot deployment on physical systems (Kim et al., 2024, Grillotti et al., 26 Aug 2025, Cathomen et al., 27 Aug 2025).
• Offline and Imitation Constraints: In offline RL or data-limited settings, USD can be combined with imitation-constraint objectives (e.g., KL-regularization to expert state marginal), Fenchel duality, and population matching for skill diversity (Vlastelica et al., 2023).

5. Empirical Evaluations and Metrics

Quantitative evaluation of USD algorithms draws on several metrics:

• Coverage and Diversity: State-space coverage, represented by counts of discretized cells (e.g., $(x, y)$ bins visited by skills), and mean or range of per-skill endpoint statistics, measure how widely and diversely the skills span the environment (Imagawa et al., 2023, Kamienny et al., 2021, Liu et al., 2023).
• Skill Discriminability and Disentanglement: Discriminator accuracy, cosine similarity between skill and state features, or DCI metrics (disentanglement, completeness, informativeness) are used to verify disentangled and distinctive skills (Hu et al., 2024).
• Hierarchical and Zero-Shot Adaptation: Agents leveraging USD, either in hierarchical RL or zero-shot goal-following, are evaluated for learning speed, coverage, and accuracy on downstream tasks. Representations learned via dynamic and factorized skills often translate to dramatic gains in adaptation efficiency (Park et al., 2022, Cathomen et al., 27 Aug 2025, Atanassov et al., 2024).
• Physical and Safety Outcomes: For robot deployment, metrics include illegal contact rates, trajectory error, success in damage adaptation, and ability to execute specific gaits or target reachability without further tuning (Grillotti et al., 26 Aug 2025, Atanassov et al., 2024, Cathomen et al., 27 Aug 2025).

6. Limitations, Open Challenges, and Future Directions

Several challenges persist in USD research:

• Scalability and Sample Efficiency: High-dimensional state or skill spaces demand efficient exploration; regret-aware generators and population-based or contrastive ensemble approaches improve efficiency but scaling remains a challenge (Zhang et al., 26 Jun 2025, Bai et al., 2024).
• Automatic Factor Discovery: Most structured approaches require prior knowledge of controllable factors; discovering these automatically, especially from pixels or unstructured inputs, is an open area (Hosseini et al., 2 Feb 2026).
• Disentanglement versus Coverage Trade-offs: Balancing skill discriminability, disentanglement, and state coverage under computational and algorithmic constraints requires dynamic or learned weighting, advanced regularizers, or scheduling (Liu et al., 2023).
• Generalization to Out-of-Distribution and Real Domains: Transferring skills robustly from simulation (or from limited, domain-specific data) to real-world deployments, and generalizing to unforeseen safety hazards or morphological changes, is critical for broad robotic impact (Grillotti et al., 26 Aug 2025, Atanassov et al., 2024, Kim et al., 2024).
• Integration with Human Guidance and Constraints: Incorporating action-free human video, natural language, or instruction-derived metrics to guide or constrain skill acquisition is a promising direction (Kim et al., 2024).

7. Representative Advances and Comparative Summary

<table> <thead> <tr> <th>Method</th> <th>Key Innovations</th> <th>Empirical Domain/Result</th> </tr> </thead> <tbody> <tr> <td>DISCS<br>(Imagawa et al., 2023)</td> <td>Maximizes MI with continuous skills on a sphere (vMF discriminator), HIPPS for sample efficiency</td> <td>Ant robot; occupies more cells with smoother, richer behaviors vs. DIAYN/VISR</td> </tr> <tr> <td>LSD<br>(Park et al., 2022)</td> <td>Lipschitz-constrained state embedding, distance-maximizing (“far-reaching” skills)</td> <td>Locomotion/manipulation; >20–50% higher coverage than MI-based USD</td> </tr> <tr> <td>Divide, Discover, Deploy<br>(Cathomen et al., 27 Aug 2025)</td> <td>Factorizes state/skill, per-factor USD, symmetry priors, style safety factors</td> <td>ANYmal-D quadruped; interpretable, safe skills deployable zero-shot to hardware</td> </tr> <tr> <td>DUSDi/SUSD<br>(Hu et al., 2024, Hosseini et al., 2 Feb 2026)</td> <td>Disentangled, factorized skill and reward structures, value factorization</td> <td>Multi-agent/kitchen; outperforms in hierarchical downstream tasks</td> </tr> <tr> <td>GISD<br>(Chang et al., 20 Jan 2026)</td> <td>Group-invariant dual, Fourier parameterization, symmetry-constrained skill discovery</td> <td>Ant/quadruped; 15–21% higher coverage, faster convergence</td> </tr> <tr> <td>CeSD<br>(Bai et al., 2024)</td> <td>Ensemble of skills, partitioned state coverage, entropy-constraint regularization</td> <td>Maze/URLB; achieves 91% IQM vs. 75% for CIC</td> </tr> <tr> <td>Constrained Skill Discovery<br>(Atanassov et al., 2024)</td> <td>Norm-matching for skill-to-state mapping, Euclidean constraint, robotic deployment</td> <td>ANYmal; achieves full-disk coverage, robust zero-shot reachability on hardware</td> </tr> <tr> <td>Regret-aware Optimization<br>(Zhang et al., 26 Jun 2025)</td> <td>Adversarial skill-generator, min–max regret, population of generators</td> <td>Ant/kitchen; up to 15% higher zero-shot improvement in high-dim settings</td> </tr> <tr> <td>DoDont<br>(Kim et al., 2024)</td> <td>Instruction-based reward shaping via “Do”/“Don’t” videos, metric learning</td> <td>Efficiently avoids unsafe behaviors; learns complex locomotion and manipulation</td> </tr> </tbody> </table>

Research in USD continues to rapidly progress towards methods that not only maximize state coverage and skill diversity, but also satisfy practical requirements of disentanglement, safety, interpretability, and real-world deployability. Recent advances demonstrate that explicit structure—factorization, symmetry, dynamic focusing—and learned or human-guided constraints markedly enhance the breadth and utility of the discovered skill sets.