Hierarchical Skill Codebooks (SPECI)

• Hierarchical skill codebooks are structured repositories of abstract skills and symbolic state representations that facilitate efficient planning and continual learning in complex sequential domains.
• SPECI integrates neural-realized codebooks with a hierarchical composition and dynamic expansion mechanism, yielding significant improvements in forward transfer and reduction of negative backward transfer.
• The framework employs attention-driven skill retrieval and mode approximation to enable robust knowledge transfer and scalable lifelong robot manipulation.

Hierarchical skill codebooks are structured repositories of abstracted, reusable skills and their corresponding symbolic state representations, facilitating efficient planning, robust continual learning, and effective knowledge transfer in high-dimensional sequential decision-making domains. Recent frameworks, notably SPECI (Skill Prompts-based HiErarchical Continual Imitation Learning), combine neural skill codebooks with hierarchical composition and dynamic expansion within lifelong robot manipulation regimes. Earlier conceptualizations include the skill–symbol loop for abstraction hierarchies, which explicitly iterates skill discovery and state abstraction in Markov Decision Processes to enable tractable high-level planning. This article provides a comprehensive account of hierarchical skill codebooks, emphasizing architectural details, knowledge transfer mechanisms, and connections to both imitation learning and model-based reinforcement learning paradigms.

1. Architectural Foundations of Hierarchical Skill Codebooks

A hierarchical skill codebook comprises structured, multi-level representations of abstract, temporally extended actions (“skills”) alongside symbolic or vector-space state abstractions. SPECI (Xu et al., 22 Apr 2025) instantiates this in a neural-realized continual imitation learning system via the following pipeline:

• Multimodal Perception & Fusion: State encoding integrates heterogeneous sensory data—tokenized language goals (CLIP text encoder plus MLP projection), dual-stream visual observations (wrist and workspace cameras using ResNet-18 backbones, with FiLM layers injecting language features for goal conditioning), and robot proprioception (joint angles, gripper state via MLP). The fused representations yield an embedding sequence, $s^e_t \in \mathbb{R}^{B\times L\times d}$, through a temporal transformer encoder.
• High-Level Skill Inference: This module queries a dynamic, expandable skill codebook to retrieve and synthesize latent skill representations. Specifically, the codebook consists of $m$ skills, each comprising key and value prefixes for transformer decoders. An attention-driven retrieval mechanism combines skill vectors into $\tilde{p}_t$, furnishing prefix-tuning parameters for the downstream latent skill generation: $z_t \in \mathbb{R}^{B\times d}$.
• Low-Level Action Execution: Temporal sequences of latent skill vectors are decoded by a second transformer, the output of which parameterizes a Gaussian Mixture Model (GMM) policy head. Behavioral cloning loss $\mathcal{L}_\text{GMM}$ drives training via negative log-likelihood of demonstrated actions.

Earlier hierarchical frameworks, as in the skill–symbol loop (Konidaris, 2015), alternate between option discovery (temporally extended skills) and representation abstraction phases, constructing multiple MDP levels linked by codebooks of abstract skills and propositional state symbols.

2. Codebook Structure, Initialization, and Expansion

The key mathematical structure for a hierarchical skill codebook in SPECI is as follows:

$$P \in \mathbb{R}^{m \times 2 \times d}, \quad K \in \mathbb{R}^{m \times d}, \quad A \in \mathbb{R}^{m \times d}$$

Here, $m = k \cdot M$ for $k$ tasks and $M$ new skills per task, and $d$ is the embedding dimension. Each skill $p_i$ provides both a key-prefix $p_{i,K}$ and value-prefix $p_{i,V}$.

Initialization & Expansion:

• For every new task $k$, the first $(k-1)M$ codewords are frozen, and $M$ new skill vectors are initialized (e.g., $p_i \sim \mathcal{N}(0, \sigma^2 I_d)$).
• The codebook grows linearly with tasks; there is no explicit skill clustering or merging, which mitigates catastrophic forgetting via expansion rather than overwriting.

Orthonormalization:

• Prior to each task, one step of Schmidt orthogonalization is applied (across $p_i$, $k_i$, $a_i$), ensuring that new skill subspaces are decorrelated from existing ones, regularizing the codebook.

This design contrasts with classical symbolic codebooks where skills (options) and abstract state predicates are constructed discretely at each MDP hierarchy level (Konidaris, 2015).

3. Skill Acquisition and Reuse Dynamics

SPECI utilizes an attention-driven mechanism for skill selection and reuse:

• Affinity Computation: For each skill, raw affinities $\alpha_i$ are determined as cosine similarities between the element-wise product $(s^e_t \odot a_i)$ and key $k_i$:

$$\alpha_i = \gamma\bigl( (s^e_t \odot a_i),\,k_i \bigr) = \frac{(s^e_t\odot a_i)\cdot k_i}{\|(s^e_t\odot a_i)\|\|k_i\|}$$

• Weighted Skill Composition: The top-$C$ skill vectors (by affinity) are selected and combined via softmax-normalized weights:

$$\tilde{p}_t = \sum_{j=1}^C \alpha_{i_j} p_{i_j}, \quad p_{i_j} \in \mathbb{R}^{2 \times d}$$

• Learning Regime: Skill vectors are learned end-to-end using behavioral cloning (GMM policy loss), without auxiliary clustering or regularization losses. Orthonormalization and codeword freezing serve as the only explicit regularization.

The skill–symbol loop formalizes skill acquisition as option discovery at each MDP abstraction level, paired with construction of new symbolic representations (“symbols”), with each phase yielding entries in behavioral and state codebooks (Konidaris, 2015).

4. Task-Level Transfer via Mode Approximation

SPECI's mode approximation module enables enhanced knowledge transfer across tasks by decomposing transformer attention weights with a learnable, low-rank, task-specific additive tensor:

$$W_k = \sum_{r=1}^R \lambda^k_r (u_r \circ v_r \circ q_r)$$

with $u_r, v_r \in \mathbb{R}^d$ global shared factors, $q_r \in \mathbb{R}^N$ task-specific mode factors, and $\lambda^k \in \mathbb{R}^R$ scaling coefficients. For an input $X^k$, attention outputs are:

$$H^k = \mathcal{W}^o X^k + \left(\sum_{r=1}^R \lambda^k_r (u_r \circ v_r \circ q_r)\right) X^k$$

This enables both skill inference and action execution decoders to maintain a shared backbone with injected task-specific variations, facilitating efficient task adaptation and mitigating negative backward transfer.

This suggests future directions may include joint, online optimization of both skill codebooks and mode parameters as new tasks are encountered.

5. Hierarchical Codebook Construction in Symbolic Abstraction

The “skill–symbol loop” formalism (Konidaris, 2015) provides a principled methodology for codebook construction in hierarchical reinforcement learning:

• Alternating Phases: Each iteration consists of (a) skill (option) acquisition and (b) representation abstraction based on the acquired skills.
• Abstraction Operator: Symbolic states at each hierarchy level are defined by constructing propositional symbols (predicates) for skill initiation and effect sets, yielding abstract MDPs with option-induced transition and reward models.
• Hierarchy Assembly: The result is a stack of MDPs: $M_0 \to M_1 \to ... \to M_n$, each paired with codebooks of abstract skills ($\mathcal{O}_j$) and state predicates ($\Sigma_j$).

Empirical analysis in the Taxi domain shows that planning can be massively accelerated (e.g., from $\sim$1400 ms to $<1$ ms) when goals are specified at higher abstraction levels enabled by codebooks of reusable options and symbols.

A plausible implication is that the codebook formalism in SPECI could be extended to use information-theoretic or distribution-driven criteria for abstraction and skill discovery, adaptively controlling hierarchy depth to optimize planning complexity over target task distributions.

6. Training Objectives and Evaluation Metrics

The dominant training objective in SPECI is behavioral cloning:

$$J_\text{BC}(\pi) = \frac{1}{k}\sum_{j=1}^k \mathbb{E}_{(s^j_t,a^j_t)\sim D^j} [-\log\,\pi(a^j_t\mid s^j_t,l^j)] = \frac{1}{k}\sum_{j=1}^k \mathcal{L}_\text{GMM}^j$$

No additional codebook-specific losses are needed beyond weight decay; structural regularization is achieved via codebook freezing and orthogonalization.

Knowledge transfer metrics:

• Forward Transfer (FWT)
• Negative Backward Transfer (NBT)
• Area Under Curve (AUC)

These are defined as in the LIBERO benchmark suite. Ablations on LIBERO-OBJECT / LIBERO-GOAL show:

Model Variant	FWT	NBT	AUC
ResNet-T, no codebook	0.60/0.63	0.17/0.06	0.60/0.75
+Codebook only	0.71/0.75	0.04/0.01	0.72/0.82
Full SPECI (codebook+mode+hierarchy)	0.81/0.81	-0.01/-0.01	0.85/0.87

The expandable skill codebook alone yields an $\sim$18% FWT gain, $\sim$75% NBT reduction, and $\sim$10% AUC rise, demonstrating the mechanism’s direct quantitative impact (Xu et al., 22 Apr 2025).

7. Interpretive Connections and Extensions

Hierarchical skill codebooks, as unified in SPECI and the skill–symbol loop, advance continual learning and hierarchical planning by:

• Automating skill abstraction and compositional reuse in neural architectures.
• Enabling dynamic expansion and freezing to mitigate catastrophic forgetting.
• Supporting bidirectional knowledge transfer via both soft codebook-based skill retrieval and task-mode adaptation.

Potential future extensions include distribution-driven skill discovery to minimize average planning cost, information-theoretic symbol selection for minimal state abstraction, and adaptive hierarchy depth to balance planning efficiency with representational overhead (Konidaris, 2015).

In summary, hierarchical skill codebooks offer a framework for scalable, compositional intelligence—encompassing both neural and symbolically structured regimes—that supports lifelong learning and real-time task adaptation in complex sequential domains.